Memory Consumption Keeps Growing

External developers... This forum's for you!

Memory Consumption Keeps Growing

Postby sctell » Sat Dec 10, 2011 4:10 am

Hi,

I have been messing about accessing fields laid out in a table matrix from an external that contains the table data.

I have found that when calling the routines below that the memory footprint keeps growing as though something is not being released within SC.

It seems related to this call.

SetFieldByName( paramPtr, yorn, theName, theString )

This is called using the following routines:

Code: Select all
-(void)displayData:(SCParamBlock*)pBlock
{
   int tStartRec = [[pBlock parameterAtIndex:2]intValue];
   int tRow;   
   [pBlock sendSCMessage:@"lock screen"];
   for(tRow = 0;tRow < pNumOfRows;tRow++)
   {
      [self displayRow:tRow withRec:tRow + tStartRec withPBlock:pBlock];
      tStartRec++;
   }
   [pBlock sendSCMessage:@"unlock screen"];
}

-(void)displayRow:(int)tRow withRec:(int)tRec withPBlock:(SCParamBlock*)pBlock
{
   int tCol;
   for(tCol = 0; tCol < pNumOfCols;tCol++)
   {
      [pBlock setValue:[[pRecordData objectAtIndex:tRec] objectForKey:[pColTitles objectAtIndex:tCol]] forFieldNamed:[pFieldNames objectAtIndex:(tRow * pNumOfCols) + tCol] onCardLayer:YES];
   }
}


These routines interface with Uli's SCparamBlock class through this routine:

Code: Select all
-(void)setValue: (NSString*)val forFieldNamed: (NSString*)fieldName onCardLayer: (BOOL)yorn
{
   Str255 theName;
   
   [SCParamBlock setStringPtr: theName toString: fieldName];
   Handle   theString = [SCParamBlock stringHandleFromString: val];
   if( theString )
   {
      SetFieldByName( paramPtr, yorn, theName, theString );
      DisposeHandle(theString);
   }
   
   if( paramPtr->result != xresSucc )
      [NSException raise: @"SCParamBlockNoSuchFieldException" format: @"No %s field with name \"%@\" exists.", yorn?"card":"background", fieldName];
}


Does anybody have any ideas please.


Thanks

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Sat Dec 10, 2011 12:49 pm

Hi,

I have been monitoring SC while my project is running with Instruments and attached is a screen grab. The memory consumed keeps growing as I call into my external and this screen grab seems to point towards createwerec although my understanding of Instruments is limited. The memory allocations for createwerec increase as i call my external and do not seem to get released.

I can't show a larger screen grab as there is a limit of 65KB.

I hope someone can point me in the right direction.

Thanks

Terry


Screen Shot 2011-12-10 at 20.44.32.png
Screen Shot 2011-12-10 at 20.44.32.png (57.34 KiB) Viewed 13121 times
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Sun Dec 11, 2011 1:10 am

Hi,

Tried another method of putting data into a field from an external:

Code: Select all
NSString* tMessage = [[NSString alloc ]initWithFormat:@"put \"%@\" into cd fld \"%@\"",[[pRecordData objectAtIndex:tRec] objectForKey:[pColTitles objectAtIndex:tCol]],[pFieldNames objectAtIndex:(tRow * pNumOfCols) + tCol]];

[pBlock sendSCMessage:tMessage];

[tMessage release];


This also works but again the memory consumption of SC increases with each call.
This time however the memory increase is only a fraction of that with the other method probably in the order of 5% but nevertheless it still increases.

This uses toolbox call SendCardMessage( paramPtr, theMessage );

Any ideas or is SC just leaking all over the place?


All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Sun Dec 11, 2011 1:37 am

Hi again,

Just repeated my external code using SC directly (no external) using this code:

Code: Select all
on displayData tStartRec
global gCellNames
put the uArray of me into tArray
set the itemDelimiter to tab
lock screen
repeat with tRow = 1 to the uNumOfRows of me
put line tStartRec of tArray into tRec
repeat with tCol = 1 to the uNumOfCols of me
put the uTableName of me & "," & tRow & "," & tCol into tCellName
put item tCol of tRec into cd fld tCellName
end repeat
add 1 to tStartRec
end repeat
unlock screen
end displayData


Because it includes basically the same "put" code as the external it is also showing an increase in memory consumption as you use the scrollbar at a very small amount similar to my previous post.

Regards

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Sun Dec 11, 2011 4:50 pm

sctell wrote:Just repeated my external code using SC directly (no external) using this code:

Code: Select all
on displayData tStartRec
global gCellNames
put the uArray of me into tArray
set the itemDelimiter to tab
lock screen
repeat with tRow = 1 to the uNumOfRows of me
put line tStartRec of tArray into tRec
repeat with tCol = 1 to the uNumOfCols of me
put the uTableName of me & "," & tRow & "," & tCol into tCellName
put item tCol of tRec into cd fld tCellName
end repeat
add 1 to tStartRec
end repeat
unlock screen
end displayData


Because it includes basically the same "put" code as the external it is also showing an increase in memory consumption as you use the scrollbar at a very small amount similar to my previous post.

Though your sample code is pretty useless as-is, if I set up a simple project that just puts text into a field in a loop it doesn't leak a single byte (at least not in 4.7.3 under 10.6.8 -- what OS are you testing in?).

I'm sorry to say that there *are* in fact quite a number of leaks that show up in Instruments when running SC. Most however are in the Toolbox APIs SC calls, not in SC itself. There definitely are leaks in the current shipping code though; I know this because I fixed several shortly before Scott decided to ship 4.7.3 and he didn't want to hold things up for verification since those fixes were part of a much wider-ranging set of changes that would have required another whole beta cycle.

One leak in particular that you should be aware of is that each call to a Universal binary external currently drips the number of bytes it takes to hold the external name as a CFString. This may be the issue you're seeing when you test SendCardMessage. Again this has long since been fixed, but it will show up you if you try to profile under 4.7.3 in Leaks. When I test a simple external that just calls SetFieldByName in current development builds it doesn't leak anything either.

HTH,
-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby sctell » Mon Dec 12, 2011 12:40 am

codegreen wrote:One leak in particular that you should be aware of is that each call to a Universal binary external currently drips the number of bytes it takes to hold the external name as a CFString. This may be the issue you're seeing when you test SendCardMessage. Again this has long since been fixed, but it will show up you if you try to profile under 4.7.3 in Leaks. When I test a simple external that just calls SetFieldByName in current development builds it doesn't leak anything either.


I am not sure I understand. Are you saying that although the leak has been fixed it still shows or has the leak not been fixed yet?

This could be it as I have a scrollbar changing the content of 160 fields on screen. For each movement of the scrollbar I call the external. So when I am testing by scrolling up and down it gets called many, many times.
However, I see the memory consumption growing in Activity Monitor.

Would this phantom leak show in Activity Monitor?

I am testing in 10.7.2 with 4.7.3

Thanks

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Mon Dec 12, 2011 6:51 am

Hi,

Here is another image from Instruments.

This relates to SetFieldByName( paramPtr, yorn, theName, theString );

Screen Shot 2011-12-12 at 14.35.31.png
Screen Shot 2011-12-12 at 14.35.31.png (41.46 KiB) Viewed 13097 times


I have used my project/external extensively to demonstrate this issue and Activity Monitor is saying it is consuming 718.4 mb real memory.

Instruments seems to be saying WENew (Waste Edit I presume) and QuickDraw call NewRgn have been called 50417 times an on each occasion 64 bytes has been allocated.

Are you saying that Instruments and Activity monitor are both incorrect?

I also tried a command line utility called top and that coincided with Activity Monitor.

All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Mon Dec 12, 2011 8:21 am

sctell wrote:[I am not sure I understand. Are you saying that although the leak has been fixed it still shows or has the leak not been fixed yet?

The fix has not yet shipped.

This could be it as I have a scrollbar changing the content of 160 fields on screen. For each movement of the scrollbar I call the external. So when I am testing by scrolling up and down it gets called many, many times.
However, I see the memory consumption growing in Activity Monitor.

Would this phantom leak show in Activity Monitor?

Definitely.

I am testing in 10.7.2 with 4.7.3
<snip>
Instruments seems to be saying WENew (Waste Edit I presume) and QuickDraw call NewRgn have been called 50417 times an on each occasion 64 bytes has been allocated.

Unfortunately (although it wasn't documented anywhere, since AFAICT Apple no longer documents changes to Carbon) the memory manager changed in fundamental ways in Lion that will break any code which depends on vicissitudes of pre-Lion implementations. So hearing that there are additional leaks under Lion (especially in WASTE, which rides the memory mgr pretty hard) that I didn't find back before Lion shipped doesn't really surprise me. When even QuickDraw itself still leaks like a sieve three releases into a new OS due to undocumented changes to underlying APIs, how is anyone else supposed to do better?

I'll take a look and let you know what I find.

Thanks
-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby codegreen » Mon Dec 12, 2011 5:13 pm

codegreen wrote:I'll take a look and let you know what I find.

I repeated both tests (i.e., putting text into a scrolling field using both 'put' and 'SetFieldByName') under 10.7.2, and Instruments reported no leaks in WENew.

Any chance you could send me a (simple) sample project that reproduces the issue at your end?

Thanks,
-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby sctell » Tue Dec 13, 2011 12:25 pm

Hi Mark,

Sent sample project and source code for the external to hello@supercard.us.

I did file a bug but it would not let me attach the project.


All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Tue Dec 13, 2011 5:43 pm

sctell wrote:Sent sample project and source code for the external to hello@supercard.us.

I downloaded and ran your sample, but I'm still not seeing any leaks. Hmmmm.

What model of Mac are you running this stuff on?

Thanks,
-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby sctell » Wed Dec 14, 2011 1:09 am

15" MacBook pro 2.4 core 2 duo late 2008 4GB


Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Wed Dec 14, 2011 1:13 am

codegreen wrote:I downloaded and ran your sample, but I'm still not seeing any leaks. Hmmmm.



Do you mean.

1. The memory consumption in activity monitor does not rise by mb's when you scroll?

2. You do not see the allocations growing in Instruments?

How are you checking for leaks?

Thanks

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Wed Dec 14, 2011 2:18 am

Hi Mark,

I have just recorded a screen capture movie, quite small to keep it under my site limit but it does demonstrate what I am seeing in activity monitor.

SC is highlighted in Activity Monitor on the left and the project is on the right. I am scrolling with the mouse scroll wheel.

If you are interested, it can be found here. Tried uploading above but forum says it will not allow .zip.

http://www.theaford.co.uk/uploads/6/0/9/2/6092496/weleak.mov.zip

All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Wed Dec 14, 2011 6:23 am

Hi Terry,

Okay, lets back up a bit here.

You probably already know this, but IIRC Leaks (in Instruments) doesn't detect what humans would call leaks, it only flags 'orphaned' blocks (i.e., allocations to which it can no longer find a pointer -- which doesn't always indicate an actual bug). Just because code is allocating blocks and never disposing of them doesn't by itself constitute a 'leak' according to Leaks.

What I'm NOT seeing is those reports in Leaks you posted of SC having such 'orphaned' allocations.

I certainly AM seeing SC's overall memory allocation grow each time I click the btn in your test project. However the allocations that are being made and not released during these cycles are not AFAICT being made by SC's code.

One place they're coming from seems to be the [pRecordData retain] at the top of your setCellData routine. Another is that the NSStrings you're creating with stringWithFormat in setTableParams never seem to be fully released. A third is [NSCFString _newSubstringWithRange:zone:], which is down inside the Foundation class somewhere, but Instruments won't tell me the call stack so I don't know how you're getting there and thus can't tell whose fault that really is.

For testing purposes I've compiled a special build of 4.7.3 which has bug fixes for the two memory leaks in our XCMD calling code and alert dialogs (just so I don't have a zillion spurious leak reports to sift through). Presumably if one stepped through your external and observed allocations after each step (which I just don't have time to do right now) one could identify where these mystery CFStrings are being created, which might reveal more about why it's happening. It's entirely possible that this is just another Apple bug, but I don't want to jump to conclusions.

-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby sctell » Thu Dec 15, 2011 4:18 am

Thanks for your comments.

I have revisited my code and I do not think there is anything incorrect, so I must conclude that the issues are elsewhere.

I obviously cannot complete an external that when working with SC/OSX displays these memory issues, so this one is destined for the trash.

All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Mon Dec 19, 2011 1:55 pm

sctell wrote:I have revisited my code and I do not think there is anything incorrect, so I must conclude that the issues are elsewhere.

I took the liberty of slapping together a pure Carbon equivalent to your THTable xthingie 'cuz I was curious about the memory issue. In the process of testing it I discovered another minor leak, this time in the code which resolves name-based descriptors (basically with a couple of object types the name is leaking in 4.7). It was a very convoluted bug, and given how small the leak is I'd probably never have noticed it otherwise, so thanks for holding my feet to the fire...

Other than that though, nothing else seems to be dripping in my version (aside from those two leaks in SC that I described earlier).

Though still roughly as incomplete as your Cocoa version, so far it seems substantially more efficient and less memory-intensive. I'd be happy to send you the code if you're interested in looking it over and fleshing it out some more.

HTH,
-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby sctell » Mon Dec 19, 2011 2:33 pm

Hi Mark,

Please send it along and I will see how I go with this Carbon thingy (is this Carbon thingy something new ;) )

I will still have the leaks though?

All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Mon Dec 19, 2011 4:05 pm

sctell wrote:I will still have the leaks though?

There are three leaks I know of in 4.7.3 which you will probably encounter using this external:

- calling a universal binary external leaks the number of bytes in the external's name
- using a name-based object descriptor for a fld, btn, grc, or menu item leaks the number of bytes in the descriptor
- the alert dialogs leak the number of bytes required to hold their text as CFStrings

Given how small these leaks are, unless you were specifically looking for them you'd need to wrap the responsible operation(s) up in a loop run hundreds or thousands of times before they'd start to emerge from the background noise of normal heap thrashing.

Two of these were the result of workarounds for early Carbon Toolbox bugs that apparently have long since been fixed, and the third happened during changes made to support long filenames and object descriptors. That code passes around lots of data structures in which the name/descriptor once was stored in a pstring but is now held in a variable-length block that's allocated separately and must be explicitly released when the interpreter is done with the object reference, and because of how this old code was designed there's no convenient central place to do it. Unfortunately there were no other such blocks in the relevant structs so I had no related code to use as a guide, and I just missed a few spots.

Aside from these three (which I built a special version of 4.73 containing fixes for to test with here) I don't see any increases in allocated memory while scrolling around your sample project using this version of the external.

Part of the 'leakage' you were experiencing was certainly due to the accumulation of undo objects in each field's WERec (which this code shows you how to avoid), but most of the memory was going up in smoke someplace else -- most likely due to unbalanced retains and releases across calls to the external. I didn't have the patience to waste more than a few minutes rummaging through it looking for the leaks though -- it quickly became clear that it would be easier to just avoid the extra overhead and complexity of opaque managed objects (which can be such a pain when you're working with one foot in each world like this) and just do everything the old fashioned (and blessedly un-mysterious) way...

Is your email addy still at btopenworld, or should I just forward it via Scott?

-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm

Re: Memory Consumption Keeps Growing

Postby sctell » Tue Dec 20, 2011 12:29 am

Mark,

I have sent you a PM with my email address.


Thanks

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby sctell » Tue Dec 20, 2011 7:27 am

Mark,

Thanks for that.

I have compiled it and run it and my first impressions are:

Memory consumption seems steady.

The speed of scrolling is very similar to ListMaster despite all the card fields.

The next step is to understand what you have done and then put in the functionality I was after.

If it turns out well I will post it in the externals area.

I think the most difficult part will be deciding the best way of implementing the selection routines that change the colour of display.

Mike Yenco, with his scripted table does not change the colour of the data field rather he adjusts the visibility of a bg grc.
The easiest would be to change the field colour but when I tried this via. scripting it affected the scrolling speed significantly.

What do you think, is the external likely to be fast enough to just change the colour of the field/text or will it be necessary to implement Mike's solution from the externals display routine.


All the best

Terry
sctell
 
Posts: 1143
Joined: Sun Jul 06, 2008 10:41 am

Re: Memory Consumption Keeps Growing

Postby codegreen » Tue Dec 20, 2011 12:29 pm

sctell wrote:Mike Yenco, with his scripted table does not change the colour of the data field rather he adjusts the visibility of a bg grc.
The easiest would be to change the field colour but when I tried this via. scripting it affected the scrolling speed significantly.

Mike's solution has the virtue of simplicity, and is definitely worth exploring. It's probably not the only way to skin the cat though...

What do you think, is the external likely to be fast enough to just change the colour of the field/text or will it be necessary to implement Mike's solution from the externals display routine.

If you use WASTE calls to set the text color performance should be fine (since this won't trigger a recalc or force the WERec to be rebuilt).

Changing the background color efficiently may be harder since SC bypasses WASTE's normal erase code (unless you just flip the relevant fields in the in-memory spot records directly). Using SuperTalk to modify the bkgnd color will probably force SC to burn down the WERec and create another (which is expensive) but you'd have to try it to confirm this. If so, I can slap something together to let you just fiddle the in-memory data directly. Since you're not looking to make changes that persist across sessions, that should be sufficient.

HTH,
-Mark
codegreen
 
Posts: 1546
Joined: Mon Jul 14, 2008 11:03 pm


Return to Xcode and the Internals Toolbox

Who is online

Users browsing this forum: No registered users and 1 guest