SQLite/Lion Issues (Disk I/O Error 10)

Having trouble using an XCmd or XFcn? Post your questions here.

SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Mon Nov 14, 2011 3:08 pm

Hi -

After several years of rock solid performance with Alec's SQLite XFCN, I'm seeing sporadic "SQLite Error 10 (Disk I/O Error)" errors running under Lion (4.7.3). These errors appear to cause the connection to the database to be lost (or at least access to a table in the database to be lost). Fortunately I have not seen any data loss so far.

Any one else experience this? Or better still have a fix?

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Scott » Thu Nov 17, 2011 2:57 pm

Jonathan.F wrote:Any one else experience this?


Nope. Not here, nor from other folks that I know that use this.
User avatar
Scott
Site Admin
 
Posts: 1217
Joined: Sat Jul 05, 2008 1:37 pm
Location: Northern Sierra Nevada

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Mike » Mon Nov 21, 2011 9:06 pm

Not seeing it here either... and I'm using SQLite in five SC-based apps that get rather frequent use and also testing in five projects that are major upgrades to those five apps.
Mike Yenco
YENCO.COM
User avatar
Mike
 
Posts: 215
Joined: Sun Jul 06, 2008 8:54 am
Location: Massachusetts, USA

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Sun Nov 27, 2011 5:11 pm

I have another clue. It appears that this error occurs after the standalone application has been left open for a significant period of time and is cured by a restart of the application. This suggests a memory leak. I only seen this problem under lion using 4.7.3 and it seems to have started with the 10.7.2 update. There are mentions of a memory leak in the SQLite release history, but perhaps there is a problem somewhere else. Any good ideas about how to narrow this down?

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Sun Nov 27, 2011 5:46 pm

I just watched the shared memory size of my standalone grow by several megabytes while it was sitting idle in the background. That doesn't seem correct, but maybe there are background OS tasks occurring. Any clues?

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Tue Dec 06, 2011 12:44 pm

Another clue. It appears that this error is occurring with much greater frequency with the computer in a low free memory state. I still don't know what is happening, but the idea that memory leaks eventually lead to a low memory state where the disk i/o is impacted is a reasonable-ish guess. I'm surprised no one else is seeing this. My experience is that Lion chews through 8GB of ram with only a handful of normal apple software running (iTunes, iPhoto, mail, Safari, etc.).

Any thoughts?

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby HairyHighlandCow » Tue Dec 06, 2011 2:04 pm

In my own testing I have not found any memory leaks in the SQLite external. Are you using any other externals in your project which you have running which might have a memory leak in them?

Alec
Externals, projects and software made with SC:
www.hairyhighlandcow.net/software/SC-projects.html
User avatar
HairyHighlandCow
 
Posts: 242
Joined: Sun Jul 06, 2008 1:45 pm
Location: London, UK

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Sat Dec 17, 2011 7:18 pm

Still trying to track down this error. I did find some reported bugs that are known to generate SQLITE_IOERR, but don't know if they actually apply to my app. If Alec ever gets a chance to update the xcmd it would be great to turn on the extended error reporting.

http://www.sqlite.org/cvstrac/wiki?p=CorruptionFollowingBusyError

What I'm wondering is how large the data files your apps are generating both in terms of numbers of records and absolute size. Maybe this is a significant difference in our usage. I'm currently see data files in the 160MB+ range and have individual relations with more than 130,000 records. I don't think that this should stress SQLite, but perhaps it is pushing something else.

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby HairyHighlandCow » Sun Dec 18, 2011 3:07 am

You were having no problems with the external when it was running on Mac OS 10.6 with the same large databases?
If so, this would suggest there is an issue with Mac OS 10.7.

Thanks

Alec
Externals, projects and software made with SC:
www.hairyhighlandcow.net/software/SC-projects.html
User avatar
HairyHighlandCow
 
Posts: 242
Joined: Sun Jul 06, 2008 1:45 pm
Location: London, UK

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Sun Dec 18, 2011 5:59 am

Correct. The problems started with Lion. I can blame Lion but I can't expect people to stop using it, so I'm working to see if there is anything I can do to address/work around it. The latest thing I have tried is increasing the cache size on the theory that SQLite might be running into a memory limit and that could be causing the io error. So far this doesn't seem to make any difference. Any help would be most welcome.

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby HairyHighlandCow » Sun Dec 18, 2011 2:19 pm

I'm afraid I can't think of anything which could be the problem. I think Mike's Archive software can make very large databases, as it uses FSIterator to catalog a whole hard disk and SQLite to perform searches. He hasn't reported any problems to me with large databases and Lion.
As I've mentioned to you before, I am unable to devote time to compiling the latest version of SQLite into the SQLite external at the moment. I had a 'recipe' to do this but the last time I tried it failed, so somehing has changed and I don't have time to research into the complexities of it at the moment. If anyone reading this has the technical knowledge to do this and is willing, I am happy to share the SQLite XCode project with them. I can't guarantee that having the latest version of SQLite built into the external will fix the Disk I/O Error, however.
All the best,
Alec
Externals, projects and software made with SC:
www.hairyhighlandcow.net/software/SC-projects.html
User avatar
HairyHighlandCow
 
Posts: 242
Joined: Sun Jul 06, 2008 1:45 pm
Location: London, UK

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby marksch » Wed Dec 21, 2011 4:57 pm

Hi,

Currently, there's a discussion on the SQLite list about "Error: disk I/O error while creating an index". I don't know whether it is related, but here's an excerpt from the discussion:

Hi,

thanks for the reply.

so, I don't have TMPDIR defined in my env. what is the behaviour of
sqlite in such cases ? is there a default ?

By default it will try /var/tmp, then /usr/tmp, then /tmp.
SQLite creates temporary files there that it uses to sort
the data before creating the index. This is different from
the database journal file, which is created in the same
directory as the database file.

You say in another post that the db is only 64MB in size.
So I guess it is not too likely that you are running out
of temp space. Still, try setting TMPDIR to something
sensible anyway.

Another way to go is to run the whole thing under [strace].
Then inspect the output and see if there is an unexpected
system call failure towards the end.


Maybe this helps (maybe not).

Best,

Mark
We have time for new projects. Contact me at http://qery.us/du now and be first in line.

Biggest free icon collection ever: http://icons.qery.us
User avatar
marksch
 
Posts: 78
Joined: Fri Aug 22, 2008 4:51 am

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Fri Jan 06, 2012 2:51 pm

I'm still seeing this problem and am a loss about how to proceed. It is associated with Lion, so I'm hoping that some known bug is causing the problem and will be fixed in the next apple update (10.7.3).

What I do know is that the problem appears to be occurring inside of the SQLite. Even if the problem is actually with Lion doing something to the SQLite data file on disk, I don't get any information about the error from within SC.

If a debugging version of SQLite existed, would it be possible to determine what is causing it? I don't have any experience debugging Xthings, so I may not be asking the correct question.

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby HairyHighlandCow » Sat Jan 07, 2012 10:24 am

Hi,
I've started a new job so I have even less time to look into SQLite, apologies. However, as stated before, if anyone wants the XCode project to play around with I am happy to share it with them.
Thanks
Alec
Externals, projects and software made with SC:
www.hairyhighlandcow.net/software/SC-projects.html
User avatar
HairyHighlandCow
 
Posts: 242
Joined: Sun Jul 06, 2008 1:45 pm
Location: London, UK

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby Jonathan.F » Mon Jan 09, 2012 9:32 am

Alec,

I (and I'm sure we) appreciate all you have done! I'm willing to try to figure out what is happening. I'm going to need some serious help, but hopefully some of the other xthing experts in this community will be able to provide guidance.

Can you send me the project or tell me where I can download it?

Thanks!

Jonathan
Jonathan.F
 
Posts: 45
Joined: Mon Oct 27, 2008 4:10 pm

Re: SQLite/Lion Issues (Disk I/O Error 10)

Postby HairyHighlandCow » Mon Jan 09, 2012 2:09 pm

Email sent.
Externals, projects and software made with SC:
www.hairyhighlandcow.net/software/SC-projects.html
User avatar
HairyHighlandCow
 
Posts: 242
Joined: Sun Jul 06, 2008 1:45 pm
Location: London, UK


Return to Using Externals

Who is online

Users browsing this forum: No registered users and 1 guest

cron