File size limits

[James Grubbs (Jimmy)]

Is there a limit to the size of file that LB 4.5.1 can handle? I know there is a memory cap due to windows limitations. My file packager crashes when packaging several large movie files. I cut the files up into pieces that LB seems to handle fine but it always crashes on the third movie file. The file seems to get to around 4gb in size before the crash.
the error I get is  (Runtime error: Invalid argument #2. Expected #long
).  If I package just the file it crashes on alone, there is no problem. I just now saw something interesting, It doesn't matter what movie files I pack, when the size of the file reaches 4,194,304kb it crashes. My system flakes out a little also, Like I have pushed memory limits and it practically has to restart without actually shutting down. I bet I'm doing something stoooopid but don't know what.

Error log timestamp Tuesday 04/09/19 04:45:23 PM

Runtime error: Invalid argument #2. Expected #long

Error(Exception)>>defaultAction
Error(Exception)>>activateHandler: <anUndefinedObject>
Error(Exception)>>handle
Error(Exception)>>signal
Error class(Exception class)>>signal: <'Invalid argument #2....'>
BasicProgram(Object)>>error: <'Invalid argument #2....'>
BasicProgram>>terminateRun: <anAPICallInvalidArgumentType>
[] in BasicProgram>>errorHandlerBlock
ExceptionHandler>>evaluateResponseBlock: <aBlockClosure> for: <anAPICallInvalidArgumentType>
[] in ExceptionHandler>>handle:
ProtectedFrameMarker(BlockClosure)>>setUnwind: <aBlockClosure>
BlockClosure>>invisibleEnsure: <aBlockClosure>
ExceptionHandler>>handle: <anAPICallInvalidArgumentType>
ExceptionHandler>>findHandler: <anAPICallInvalidArgumentType>
APICallInvalidArgumentType(Exception)>>activateHandler: <anExceptionHandler>
APICallInvalidArgumentType(Exception)>>handle
APICallInvalidArgumentType(Exception)>>signal
KernelDLL(DynamicLinkLibrary)>>invalidArgument
KernelDLL>>setFilePointer: <aFileHandle> offset: <4294967296> offsetHigh: <aByteArray> move: <0>
FileHandle>>readInto: <'a抪«…mm…äk%’$ž6...'> atPage: <1048576> pageSize: <4096>
FileHandle>>readInto: <'a抪«…mm…äk%’$ž6...'> atPosition: <4294967296>
File>>readBuffer: <'a抪«…mm…äk%’$ž6...'> atPosition: <4294967297>
FileStream>>position: <4294967296>
FileStream>>putBytesFrom: <'½ L\âµ_«üê;©Êr''5f...'> from: <1> to: <250000000>
FileStream(WriteStream)>>putBytesFrom: <'½ L\âµ_«üê;©Êr''5f...'>
FileStream>>nextPutAll: <'½ L\âµ_«üê;©Êr''5f...'>
BasicFile>>writeItem: <'½ L\âµ_«üê;©Êr''5f...'>
BasicFile>>writeItemCr: <'½ L\âµ_«üê;©Êr''5f...'>
[] in PrintCommand>>deviceCr:handle:
BlockClosure>>value: <aBasicProgram> value: <'#2'> value: <aBasicStringContext>
BasicTripleParameterContextHolder>>value
[] in BasicProgram>>begin
ExceptionHandler>>evaluateProtectedBlock: <aBlockClosure>
[] in ExceptionHandler>>activateDuring:
ProtectedFrameMarker(BlockClosure)>>setUnwind: <aBlockClosure>
BlockClosure>>invisibleEnsure: <aBlockClosure>
ExceptionHandler>>activateDuring: <aBlockClosure>
ExceptionHandler class>>handle: <anError class> with: <aBlockClosure> during: <aBlockClosure>
BlockClosure>>on: <anError class> do: <aBlockClosure>
BasicProgram>>begin
BasicProgram>>gotoAndIfStoppedBegin: <'[createPackage]'>
BasicProgram>>handlerName: <'[createPackage]'> evaluate: <aBlockClosure> callParameters: <anOrderedCollection>
BasicProgram>>submitHandlerName: <'[createPackage]'> evaluate: <aBlockClosure> callParameters: <anOrderedCollection>
BasicProgram>>submitHandlerName: <'[createPackage]'> callParameters: <anOrderedCollection>
LibButton>>click
BasicGraphicDevice>>click: <aLibButton>
LibButton(Button)>>click: <aLibButton>
Message>>perform
Message>>evaluate
LibButton(Object)>>triggerEvent: <#clicked>

[Chris Iverson]

Oh, I'm sorry, I thought I posted this already. When I saw you making the file packer, the first thing I tested with it was files >4GB, because those are indeed problematic in LB.


The problem is that LB internally uses a 32-bit integer to track the file position(which gets updated every time you read/write to a file). That means it can only address 32 bits worth of byte count, 2^32 = 4294967296, or exactly 4GB.

If you require working with files larger 4GB and larger, you'll need to switch over to using the Win32 File API directly.

I'm gonna see if I have anything on hand already to demonstrate doing so.

[James Grubbs (Jimmy)]

I would just hate for it to crash when someone I might not even know is using it. I could give a disclaimer. I doubt my purposes will ever require it. I was just doing some additional testing and that's when I saw the issue. Thanks Chris, If you find anything, that would be awesome.

[Mark Dunham]

Apr 9, 2019 22:47:08 GMT -5 James Grubbs (Jimmy) said:

I would just hate for it to crash when someone I might not even know is using it. I could give a disclaimer. I doubt my purposes will ever require it. I was just doing some additional testing and that's when I saw the issue. Thanks Chris, If you find anything, that would be awesome.

You could also build in a check for the size of file and give a dialog letting users know files greater than 4 GB are not supported at this time.

[Rod]

Well point is you can't check the file size because files over 4Gb will give false readings because lof() can't measure >4Gb it reads files size mod 4Gb and you get the remainder returned. So a 32bit system is limited to lower file sizes. Not just a Liberty issue.

[tsh73]

you can't check the file size

I wonder if filesize returned by FILES
(

info$(1, 1) - the file size

)
is valid.

Cannot test because I have no such files (this netbook has just 32GB SSD ;p )

[James Grubbs (Jimmy)]

Could tally the size of each file and when it reaches a certain point, give a limit message but Rod is correct, if someone selected a file greater than 4gb, it would be a problem.

[tsh73]

Just created big file (copy /b file+file+file+file dest, from 1.5Gb file)
Got val(info$(i,1))
6256877568
(DIR shows 6 256 877 568)
So FILES return right size even of > 4Gb.