Import upgrade

New feature: http://www.librarything.com/blog/2008/05/import-upgrade.php

Note: The next few messages relate to the fact that we delayed posting details as we troubeshooted it on the live site, and there was some confusion between this and another feature we're working on.

Is this just for the import or for the other parts (hash collision, ISBN separation) too?

That's separate—and coming up. Two upgrades in one night? Will you faint?

In that case, I don't think you actually announced the Import Upgrade new feature.

Right, I didn't. But I will now—fifteen minutes before we go down for the other upgrade...

we troubleshooted it on the live site

troubleshot??

Kabloooooom!

Would this/these new update(s) include finishing the other authors, so I can fix the books I entered before the new author functions came out?

Edited for decafinated spelling.

No, I'm waiting for Casey to have some free time to get that out.

The blog post shows a screen where you can sync your import with your current data and has checkboxes for things like "replace reviews with imported info." Will this overwrite your review on LT if the same book being imported has a review also? If not, could you add an option to import reviews etc. into LT but only if the data isn't already there? Perhaps for tags you could add the option to append the imported tags to the current tags.

9> Perhaps you could make this a priority since Casey hasn't been able to get it out on his free time for several months now. Are you sure he has any free time?

Hi Tim, thanks for this new feature. Is there someplace where the format(s) for import are defined? I've got a lot of book data in a desktop cataloging app that I can export, and then import into LT --- but I need to know the expected field orders/names, etc. for the import on LT.

Thanks again for the continuous improvement to the site.

To be able to refine a database and then upload it will improve the accuracy/speed of my legacy library no end. Thank you.

-Addy

Aha! When you say, "If your file is formatted properly—formatted like the LibraryThing export or any of our competitors'—we now import non-ISBN books." -- my immediate thought was:

Is this how you envision copying the records of non-ISBN books from another LT user's library? Can I export a book from some one else's library, then import it into my own library?

Or do I need to open a Shelfari account, put the series of 135 Compleat Anachronist booklets into there (duplicating the work I've done on LibraryThing), then tell all my friends that have been either waiting for "Copy this book" to return in some incarnation, or who have been painfully copying my data field by field, that they can import it directly from there? Note that there is only 1 edition of each of the CA booklets, but the amazing variation of input styles chosen by owners has often made it look like there are many different editions of each. And since some books have many authors, and each author goes by at least two separate and distinct names, it has been impossible to fully combine the ones entered.

Tim never said that you would be able to modify your data on your machine after downloading it. So, as far as we know, it could only be used as a backup of your LT data. Previously Tim has said that he is against mass importing of people's spreadsheets and databases as that could lead to a massive amount of bad data in the system. I haven't seen anything to suggest that he has changed his mind.

>14 EowynA: What might work better for you and your friends: you export your data to a file, then email the friends who need the file, so that they can then import the file to their library.

>16 gilroy: - yes, your method is much more attractive than opening an account on another system.

But I still don't know if one can export LT data from someone else's account - I'll try that this evening. Then they can do it themselves.

Will this overwrite your review on LT if the same book being imported has a review also? If not, could you add an option to import reviews etc. into LT but only if the data isn't already there? Perhaps for tags you could add the option to append the imported tags to the current tags.

Both are good ideas. Mike, what do you think, a drop-down for each data-type?

Reviews: Ignore; Replace; Replace if no review
Tags: Ignore; Replace; Append

?

Perhaps you could make this a priority since Casey hasn't been able to get it out on his free time for several months now. Are you sure he has any free time?

I can't. The MARC code is all in Python. Unfortunately, this waits on his time. I'm sorry about it, and just as frustrated as anyone else. Unfortunately, we've got a lot of work.

>14 EowynA:

I'm sorry that the green plus doesn't work the way you want it to. We would like to change it, and soon. I think it was important for us to get an import working. It was leading to a one-way bleed because, no matter how good our cataloging is, once you used another service you couldn't "get back."

In fact, you cannot import the books into Shelfari like that because Shelfari is an ISBN and Amazon only application, and even if you could, their export does not include important fields.

At present, I have Casey who's on LTFL exclusively, Mike who's part-time and new, and myself. And that's it for code. So I'm focusing on one thing at a time. Tonight it's editions and combinations.

>18 timspalding: (timspalding)

That sounds great. Should these drop down lists replace the checkboxes or supplement them? Should be an easy improvement either way.

>14 EowynA: (EowynA)

I definitely like the idea that you could export someone else's library (or a group of books from someone's library).

>15 jjwilson61: (jjwilson61)

You can "modify the data on your machine" if you stay within the formatting rules of the system. This will require some basic documentation before it becomes completely clear which I think will appear on the initial import page or the faqs. Ultimately if it's a CSV and has fields that correspond to the fields the system is looking for everything should be fine. Polluting the system is not an issue here because if your file is detected as a LibraryThing export or one of the competitors' files we're only syncing up user-centric information. We're still getting the core data about the book based on the ISBN using the same algorithms used when adding books manually.

>12 manque: (manque)
If you do an export of your LT books and copy that format it will work fine. You just need the same fields. Order isn't important.

-Mike

I tried exporting my library as a csv yesterday and adding some tags before reimporting and resynching. However, I seemed to have a few problems.

1. I got an error message (which I'm afraid I don't have any more as this was on my work computer and I'm at home today) Something about invalid key, I think.

2. Despite finding over a thousand ISBNs it only put 15 into the queue.

3. I now have 11 new books in my library that I know nothing about, 10 without ISBN, and all having the tag "not started" which I've never used.

Is it possible that my data got mixed up with someone else's? Particularly as the import queue says that 4 failed.

20> (Scorbet)

Sorry to hear this...I'll be looking into it ASAP.

20> (Scorbet)

I've spent some time trying to trace your problems without much luck but I still have a few tricks up my sleeve. I'm unsure at what point someone else's books would get crossed over into your library. I'm hoping you could help me trace the issue by clearing your queue (if there's anything still in it) and attempt re-uploading your file. If there's still problems could you copy/paste the error message in a message to me here or in an email (mike @ librarything.com)?

I left a comment on your profile with more info/questions.

Thanks in advance for any additional feedback you can offer.

-Mike

I'm curious if the feature to designate a tag for a whole import will be restored, or has been discontinued with an import upgrade. I didn't find this option when I tried to add books today with the import method

>23 PC-Stans-Liturgy:

Thanks. I've been meaning to ask Mike to add this back. I don't think he was aware it existing—because I never told him. I'm glad to hear someone misses it.

Mike?

23/24> Sonya was asking about this today also. It's on the list... -Mike

>22 notmydadslibrary:

I've cleared the queue, but I won't be able to get at the exact file until Monday - it's on my work computer and we had an extra long weekend over here. I'll try it then though.

Thanks for looking into it.

Mostly I love it. My spreadsheet had over six thousand books, without ISBNs, natch, so I love that I'm getting them in.

My disappointment is that even though I headed my list just like my Librarything export file, a lot of stuff didn't come along. No tags, no date read, no author.

Kaethe, can you send mike your file?

mike at LibraryThing.com

T

Thanks to everyone who posted on here so far. I'm working on a number of updates and fixes. I'll post more details here shortly and the updates should trickle in over the next several days. -Mike

Update...

Fixed:

* Tags and authors were not being imported (Kaethe) if you used field names from the XLS files because the we were only supporting the field names in the CSV files. I've added support wherever possible for the XLS field names so that it will work but we recommend using the CSV export in favor of the XLS format when exporting your books to guarantee all your info is imported. I've added a test for this scenario which produces an alert on the import configuration screen so you'll be made aware of this. So you should be able to reimport and uncheck the fields that have already imported (or not) to get your tags and authors synced. Unfortunately this won't help with ISBN-less books because the import algorithm has no concept of 'duplicates' for these books. We may need to think about a workaround for this so you can easily import the same file over and over again (possibly your making changes with your some offline application or text editor, etc). I propose that if the title matches it's a duplicate and you get another set of checkboxes to configure how your books personal info in synced. Tim, what do you think?

* Issues reported on the blog by Johan where his XLS file was not imported cleanly are fixed.

* Translation issue. I missed a couple fields but translation junkies will be happy to know that they can now translate everything new on the import screens.

Issues pending:

* Date read and date added fields are currently not imported. For now you won't see an option to have these tags synced but there will be a fix soon.

* When you import reviews the tally on your profile page is not updated. Fix coming soonish.

Features planned:

* Batch tagging. Tag all the books your importing in one shot.

* Finer grained sync settings. You'll be able to do things like have the tags being imported added to your existing tags or have your info be skipped if there's already a review/note/tag for given book.

* We'll definitely add more file formats as soon as problems listed above are resolved. New formats can be added very easily so if you have one you want added please email a sample to mike @ librarything.com. This will save me a bunch of time. Thanks.

Tim, what do you think?

I think that supporting XLS is risky, as with supporting TSV. I'd rather support one format well, and broaden what we can do with it. Supporting many different formats is a recipe for stagnation and future maintenance problems.

I don't see a way to deal with re-importing non-ISBN books. *Some* of the files—just Shelfari?—include a unique ID. I suppose we could capture that. But keeping track of different sets of unique IDs doesn't sound like much fun to me.

Mike, tell me when your half of the date-read issue is done. Then I'll get mine working.

On the review numbers, I'm going to go look now. It makes no sense to me.

>review numbers

Ah, it wasn't setting the "yourrev" field. It is now. I'm running a reconciliation batch now, so it should be good soon.

T

Thank you all for making this enhancement. With the start of a long holiday weekend in the U.S. I've been able to try it out today.

So I started with a CSV export of my current LT catalog to get a format to work with. I dutifully included publication dates in the Date field, as they'd come through in the export. Upon uploading the new CSV file, the books flowed into my catalog beautifully, but all of the Dates showed up as 2008. I had to manually edit each new book and put in the correct date.

Even your ISBN books? They shouldn't have been imported per se, but re-fetched.

Hi Tim, I thought we weren't supporting XLS just because of the encoding issues of the LT XLS. I figured out how to detect the encoding and convert. We can take it out but we may want to alter the XLS export so it uses the same field names as the CSV to avoid confusion. Easy to revert back and just warn users that the XLS import is importing ISBNs only and not other metainfo fields. I have most of this evening available to take care of this and the other things we've agreed need work. -Mike

Ps. I assume this date thing is a non-ISBN related issue. Should be able to fix it rather quick.

>35 notmydadslibrary:

But we can't change the TSV because other things—PalmThing, for example—rely on it. I'd really rather kill it entirely. So I think we should at least "deprecate" it—put a sign on it that it's not the best way.

I won't be back until around 10, if you're planning oncoming over.

If "the best way" is better than the TSV, it should be easy for the rest of us to either use that or transform it into something TSV alike, so please go ahead with this. I think quite a few of us would really like to see some of the data entered for the new author roles safely exported.

Update...

23/24> (PC-Stans-Liturgy) Added mass tagging field. Now you can designate a set of tags that will apply to all books being imported.

33> (Suncat) I looked at the files you uploaded and most of them had 5/24/2008 as their publication date. Can you send me a sample of a file if you have one that doesn't import the dates properly? Thanks. My address is mike at librarything.com

So most of the situations that caused errors or unexpected results have been handled. The main items pending are the finer grained sync settings and handling date read/date added fields (details in post #31).

-Mike

>22 notmydadslibrary:

I tried it again yesterday with the same csv file and it worked perfectly. I'm still unsure where the (now deleted) books came from the first time though...

OK, I tested it out, trying to get more of my books having info from LoC, rather than Amazon. I did a test batch of 40.

Despite the fact that I know several of them are in the LoC, and in ILCSO (my second choice), they are all still reporting source:Amazon.com. Does this change if they're imported from a new source?

I'd really like to decrease the amount of my catalog that comes from Amazon (I really wish I'd know about their data quality when I started here, or that they weren't the default).

I'm a new user and have been trying to import a shortened version of my non-ISBN excel spreadsheet for a few days.

I have tried the importation with excel, csv and plain text courtesy of notepad, cutting and pasting into the import field on screen.

I am still unable to import my file, and the import tool does not recognize my titles as "books".

Furthermore, I am not sure which titles to use on the spreadsheet column headings and if this is a necessary step. I have matched the headings in the add book category, and was hoping that the comments could be imported, but no luck there either.

Suggestions please? I would be happy to just get the titles imported.

Thank you,

vintage_books

>34 timspalding: (Tim) No, the ISBN books came through just fine, as usual. But most of the books I'm cataloging right now are older, and don't have ISBNs.

>38 notmydadslibrary: (Mike) I've sent you the most recent CSV file which I used, for importing the most recently-added books in my catalog. These do seem to have the erroneous 2008 publication date, and I haven't edited that out so you can see for yourself.

What's TSV format, like in messages 36 and 37? I know what CSV is.

TSV is tab-separated values.

Vintage: You need to make the file "look" like our CSV export—us the same headings, etc. Send email to mike you have questions.

Mike, Tim, this is an impressive and useful upgrade. Thank you.

The Vox link in the green box on the right of the import page links to a server named "athena" and not to "www". Moreover, I think the s in FAQS slightly further down should not be a capital s.

Update...

40> (timepiece)

Looking into this.

41> (vintage_books)

The reason your list didn't import is because we don't support the XLS format per se. You should be able to 'Save Copy As' in Excel as a CSV file. Once you do this you can continue to edit this new file in Excel but you will have a file that will also import cleanly into LibraryThing as long as your fields are the same as ours. The fields we currently support are:

isbn
rating
review
tags
entry date
author (last, first)
publication info

NOTE: a 'dateread' field will be supported soon.
Also, these fields aren't case sensitive.

42> (Suncat)

The problem was that I wasn't handling the date field correctly. That's one that should have been caught. The easiest way to fix your batch is to delete any non-ISBN that were in the file and re-import them. Dates should import cleanly now.

45> (koffieyahoo)

Apply coffee, rinse, repeat. Fixed.

Thanks everyone, more to come...

-Mike

Mike - How do I find your email address to send you my file and questions? It will be a quicker resolve than posting to this forum.

Thanks

vintage_books

I'd normally say use the Congratulate/Complain link at the bottom of the page, but it doesn't seem to have been updated with Mike's info yet.

Try: mike at LibraryThing.com

(Tim mentioned it in Message 28)

Thank you for re-posting that bit of info.

I've sent my file over to Mike, with a request for more specific instructions so I don't have to bother him again.

vintage_books

Hey, I've just tried to import some books via a CSV file. As far as the ISBNless items, it worked great, but I've got 276 books waiting in a queue that was supposed to take 15-16 minutes and I've been waiting on it for about 40 minutes. I'm not sure if the site is just slow today or if I've got other problems going on.

Thanks,
Amanda

52> (sollocks)

Amanda, your books are being processed now. Things got way backed up.

-Mike

Cool, thank you very much. It took me a little bit of rummaging around to find out that the my queue was thrown in there w/ that of the whole site. So that wait didn't seem that big of a deal, I just wanted a more accurate indication of how much time it would take. I don't mind the wait as long as I'm expecting it.

I'm not very adventurous:)

Thanks Mike, for restoring the mass tagging field when importing books. Works great again!

Update...

Fixes added today:

1) Large files (over 2K books) were not importing correctly. Should read anything within the 2MB file upload limit now. This accounts for unexpected results reported here and via email.

2) Ugly database error experienced by some users this morning are Fixed.

3) ISBNs that have had their leading 0s stripped by another program were not importing. Fixed.

-Mike

Getting there...trying to wrap up loose ends on this project today.

Thanks for the update Mike.

Can you please confirm that you received my email with test spreadsheet for processing for whever you get a minute? I didn't receive any delivery or confirmation receipt and I hate to bug you when you are so busy.

Thanks again-

vintage_books

Hi, all... I'm trying to import my library, and every time I submit either a file or a list of ISBNs to the import page, all I get back is a mostly empty page containing the number of ISBNs the importer found. Nothing gets added to my queue, and nothing shows up on my page. Is the importer in the midst of being tweaked? Is there a site status page I could check in the future instead of pestering people here?

UPDATE: How's that for service? As soon as I posted this message, the importer started working again! Sweet!

Thanks for your time,
othiym23

57> (vintage_books)

Sorry... I was planning a response but the problems that cropped up this morning distracted me.

You need at least 3 fields to match (even if there's no data for that column) to make it recognizable as a LibraryThing file. The fields are:

rating
review
tags
entry date
author (last, first)
publication info

NOTE: a 'dateread' field will be supported soon.
Also, these fields aren't case sensitive.

Also your 'date aquired' column should be called 'entry date' if you want it to be imported. Support for importing this field not yet ready so if you do want to keep it you may want to wait a few days and keep watching here for an update that addresses this.

Thanks for your inquiry!

-Mike

58> (othiym23)

When you say it "started working again" do you mean the queue started adding your books or did you have to re-upload your file. Thanks.

If you are just waiting for the queue to process your books you can see this info on the import page: http://www.librarything.com/import

-Mike

46>

Finally had a few minutes to test. Scrapped the last seven books cataloged and re-imported with the same file. All of these were non-ISBN books. All of the publication dates came through just fine.

Thanks for fixing this.

Update...

Fixed issue when trying to upload files with leading whitespace. I'm not sure why this happens but now it's not a problem. Lines with nothing but whitespace are ignored.

Thanks to vintage_books for pointing this out!

-Mike

Is there a bug in import that allows people to import more than the 200 maximum on a free account? See posts here from post 378: http://www.librarything.com/talktopic.php?topic=19714

Mike-

Did you get my latest spreadsheet? Still unable to import....waiting for demo file to reformat to your requirements....

Thanks!

vintage_books

Update...

Thanks to vintage_books for your help and feedback that led to these changes.

* Made sample CSV file available for download (on tools and import pages). This can be used as a starting point if you want to create a spreadsheet that will be importable to LibraryThing.

* Altered algorithm so it's less picky about what constitutes a properly formatted file for import. In other words less fields have to match for non-ISBN books in the file to be importable. We'll see how this works.

* There was a problem that led to users losing their tags (thanks to Tim for magically restoring them with his ninja-like SQL skills). Tags are now always appended (until we add finer grained import control).

More to come...

Mike

Mike-
I can't find the sample CSV file. Can you please point to its location?
Thank you,
vintage_books
PS: Thanks for the kind words.

The sample file is on the Import page, on the very bottom of the green box on the right hand side.

Thank you for the location.

There appears to be a formatting error on column D, which should have all cells be "numbers" instead of a default formula.

> 68

It's a CSV file, i.e. just text. There's no inherent formatting, it's just your spreadsheet software trying to be smart. Change the formatting according to your needs and you should be fine.

> 68

When I load the file into Excel, it gives each column the default width, and column D shows 'ISBN' (in row 1), and then 8.05E+08 underneath that, since the column isn't wide enough to display the 10 digit ISBN.

Widening the column enables it to show 805061762.

Similarly, column I shows 'ENTRY DATE', and then ########. Widening that displays "17/06/2008 12:03".

Opening the file in Wordpad shows the original text in both cases: 0805061762 for the ISBN, and 2008-06-17 12:03:21 for the date. Note that, strictly speaking, the ISBN shouldn't be formatted as a number, or if it is, it should be a ten or thirteen digit number: that leading 0 is relevant.

We decided to go with square brackets around ISBNs when you export your books to avoid the leading 0s being stripped. This should help. -Mike

Wow, I have been waiting for custom import forever and can't believe I missed out on the rollout. And I also can't believe it is so badly advertised on the import page (I found the link to the blog announcement and then this page via a random Google search).

A few questions:

1) is it possible to disable the LoC/Amazon/?? checks completely? After doing my original cataloging in Readerware well before LT showed up and spending a huge amount of time sanitizing the crappy data from Amazon & co. I'd rather *not* see my manual updates squashed again (sorry, but I cannot be sold on the "high-quality library data" line LT has been peddling for all those years).

Consider that it will also lighten the load on your servers and connections with those reference sites, as I will probably re-importing over 1,000 books multiple times (I currently have slightly under 1,500 books in LT -- more if I can get this import thing working w/ ISBN-less books -- in 4 different languages from 5 different countries) and am not quite ready to trust LT as my master DB yet ;)

2) What about books with multiple authors? The LT catalog handles them, but they don't show up in the export or the sample import file.

3) Is the "publication info" field's format clearly documented somewhere? The sample has page count (which my export doesn't), my export has the edition number (not in the sample import file).

I *am* doing a (measured) happy dance in front of the computer. Thanks for finally making this feature available.

72>

1-Currently the import engine will use data from the selected libraries unless the book has no ISBN.

2-There's isn't currently a way to import multiple authors but it wouldn't be hard to add.

3-At the moment we don't do anything with the 'publication info' field. It comes from the current export engine and we use it to help identify LT files. Needs work.

I think these are all things that should be considered for the next iteration of this application. Part of what needs to happen, specifically, is the export engine needs to be brought up to date to play nicer with the import engine.

-Mike

OK, thanks for the update. Point #1 is actually blocking for me.
I can get better input from LoC than Amazon for US books, but about 20% of my collection is in French and the quality of the information from the available libraries is dismal.

Hmmmm. What happens if trying to import with ISBN but the LT importer doesn't find the ISBN in any of the selected libraries? Does it then import the basic info (author, title...) from the file (as if no ISBN had been provided), or does it just skip the entry?

There *should* be an option to do an export and re-import than has a null overall effect... hopefully sometime in the not too distant future?

74: Have you tried some of the French libraries?

74>

If it has an ISBN that can't be found in the selected libraries the book won't be imported. -Mike

I have a file to be imported into my legacy library (no ISBNs). Import always fails, regardless of format. It uses these fields:

author (last, first)
title
date
publication
your tags
comments
language 1

Is the presence of one of these field hindering the import?

The import fields have to *exactly* match those in the sample file.

>78 ablachly:
Thanks. I've rejigged it a bit, and got a not-so perfect import.

I think I'll have to continue assembling this library offline, and wait until the fields I use are fully importable.

Or do it the hard way - Add Books or import for the basic data, then cut-and-paste the rest. Or cut-and-paste on your not-so-perfect import, if you got a reasonable facsimile of the books you want. I haven't imported any of my books (3300+), I've put them in individually - it's slower than import, of course, but I have them in _now_! And having them in is incredibly useful.

BTW - did you see the notice that the title field will now take 500 characters? The full titles should fit, now.

>80 jjmcgaffey:
Manual import isn't really an option for this library; there's very, very little information that LT can supply, and keying in long titles in foreign languages is hugely slow.

18th century books often have paragraph-long titles that would make Work pages unwieldy, libraries harder to peruse, and combining awkward. I still find the Comment field most suitable for full titles

81>

What's wrong with searching libraries? I'm finding the vast majority of the books in Darwin's library are in library sources (in the correct edition, as if that needed to be said) -- I've only had to manually enter maybe 1 in 10 so far, and usually the author, year, and first few words of the title are enough to pull it up, paragraph-length title and all.

Edit: Or are you using "manual" to mean "one-at-a time"? I think my question still applies, though.

I manually imported 600 books from a software package called Logos that I own. It went quite well, except that Logos does not export the ISBN. But no worries.

My question: all book covers can be found on https://covers.logoscdn.com/ unique book identifier /cover.jpg. I can of course manually do this for 600 books. With 5 books per minute, that would take me over 2 hours. But is there a way to import this immediately when I import my books?

>1 timspalding: I just tried the link at the beginning of this message stream and I'm getting a 404 error. I'll try it again on my home computer when I get home.

>85 ulmannc: I guess they have changed the blog over the twelve years that have passed.

>86 anglemark: Duh. I didn't even look at the date. I assumed it was a "new feature" Looking at dates is like pressing the page down button to read the rest of the message. Such is life! The Electronic Luddite!

>84 nijntjepluis: Not at this time