Integrate littlefs into mbed OS #5538
Conversation
git-subtree-dir: littlefs git-subtree-split: 663e953
- Changed log statements to use the debug function - Changed %d to %ld given the type of int32_t in arm-none-eabi-gcc. In mainstream gcc this is not the case and may cause problems to upstream.
Because lookahead is stored efficiently as a bit-vector, this only requires a ram increase of 48 bytes (2.1% of benchmark), but decreases the SD benchmark runtime cost by 32 seconds (21.9% of benchmark). Note this is unimportant on devices with byte-reads such as NOR flash.
9843402 Fixed incorrect return value from lfs_file_seek 273cb7c Fixed problem with lookaheads larger than block device d9367e0 Fixed collection of multiblock directories a83b2fe Added checks for out-of-bound seeks a8fa5e6 Fixed some corner cases with paths 26dd49a Fixed issue with negative modulo with unaligned lookaheads 0982020 Fixed issue with cold-write after seek to block boundary git-subtree-dir: littlefs git-subtree-split: 9843402
- TESTS/filesystem for mbed OS filesystem APIs - TESTS/filesystem_retarget for mbed OS retargeted stdlib APIs converted: - test_dirs - test_files - test_seek - test_parallel
required intrinsics for: - lfs_ctz - lfs_npw2
454b588 Updated SPEC.md and DESIGN.md based on recent changes f3578e3 Removed clamping to block size in ctz linked-list 83d4c61 Updated copyright 539409e Refactored deduplicate/deorphan step to single deorphan step 2936514 Added atomic move using dirty tag in entry type ac9766e Added self-hosting fuzz test using littlefs-fuse 9db1a86 Added specification document git-subtree-dir: littlefs git-subtree-split: 454b588
The RBIT instruction reverses the bits of a word, not REV
- Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
All calls are blocking, so a single mutex is able to garuntee synchronization across all relevant functions.
- Removed list of warnings on signedness of integers in printf - Fixed issue with "true" in ifdef
The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
The previous math for determining if we scanned all of disk wasn't set up correctly in the lfs_mount function. If lookahead == block_count the lfs_alloc function would think we had already searched the entire disk. This is only an issue if we manage to exhaust a block on the first pass after mount, since lfs_alloc_ack resets the lookahead region into a valid state after a succesful block allocation.
|
Restarting due to rogue uvisor CI job marking this is a failure. |
Build : SUCCESSBuild number : 622 Triggering tests/morph test |
Test : SUCCESSBuild number : 444 |
Exporter Build : SUCCESSBuild number : 249 |
There was a problem hiding this comment.
Some work to be done to readme/links in readme that point at external repos that should be shut down and pointing at the release this is part of.
The logging using LFS rather than mbed highlights we still need to get that fixed. Please talk with @SenRamakri about what you've done here and make sure it can be replaced in a compatible way (for the stack implementation)
Otherwise LGTM 👍 and remind me to chat about later about assert
0xc0170
added
release-version: 5.7.0-rc1
ready for merge
needs: CI
and removed
needs: review
ready for merge
labels
Can you please update? we will retrigger tests asap. Uvisor is currently being built, device failure should be fixed now |
|
@sg-, Can update to be self contained. Is it fine to keep the external links to the FUSE and JavaScript wrappers in the related projects section? Or should I remove that section? |
geky
force-pushed
the
littlefs-staging
branch
2 times, most recently
from
13446f0 to
403ce88
Compare
geky
force-pushed
the
littlefs-staging
branch
from
403ce88 to
c613030
Compare
|
Talked with @sg- offline, he says this is fine, so we just need one last test run for sanity |
|
/morph build |
Build : SUCCESSBuild number : 631 Triggering tests/morph test |
Exporter Build : SUCCESSBuild number : 258 |
Test : SUCCESSBuild number : 455 |
|
/morph uvisor-test |
|
/morph echo |
|
I recieved your comment! Here's what you wrote!
|
What is littlefs?
littlefs is a little fail-safe filesystem designed for embedded systems.
It comes with three key features:
Bounded RAM/ROM
Here's a comparison of the memory consumption between the FAT filesystem and littlefs:
By just switching from FAT to littlefs you will have saved 13 KB of ROM.
Power-loss resilience
The main drawback right now of the FAT filesystem is that there is no protection against power failures. Unless you have constrained when the device could lose power or be reset, you are basically relying on luck to not end up with a corrupted filesystem.
The funnest way to demonstrate this is with the example program that is in littlefs's README.md. Spamming the reset button with the FAT filesystem will quickly lead to problems:
Wear leveling
To continue bashing on the FAT filesystem, here's a comparison of littlefs and FAT wearing down a block device with 3KB of static data and 3KB of dynamic data:
The wear leveling algorithm in littlefs is not perfect, but it does give you the very nice property that expanding the size of storage increases the lifetime of the device.
Here's an interactive simulation where you can see how the wear leveling behaves with different configurations:
http://littlefs.geky.net/demo.html
How does it work?
I'm already pushing the limits of what is reasonable to shove into a PR, but I would encourage you to look into littlefs's DESIGN.md if you're interested.
How do we know it works?
The littlefs has quite a few stages of testing:
A large set of unit tests run in a simulated block device on Linux. This provides coverage for forseeable issues. Any bug fixes come in with a unit test to prevent regressions.
https://github.com/geky/littlefs/tree/master/tests
The littlefs-fuse is used to hook up littlefs with the kernel. A subset of the unit tests are self-hosted on littlefs itself to provide a level of fuzz testing on the functional API.
A set of integration tests run on the mbed filesystem API on a device with a physical storage device.
https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem
A set of integration tests run on the retargeted stdlib API on a device with a physical storage device.
https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_retarget
A set of repeatable tests are ran in a simulated mbed block device with a full file-level filesystem check on every single erase cycle. This guarantees that the filesystem operations are atomic and the filesystem is never in an invalid state. (provided by @c1728p9).
https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_recovery/resilience
A set of repeatable tests are ran in a simulated mbed block device with simulated wear until the block device is exhausted. The block device is increased and the test is ran a second time. The test only passes if the increased size results in a proportional increase of filesystem lifetime. This guarantees that the properties around wear-leveling hold true. (provided by @c1728p9).
https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_recovery/wear_leveling
A set of repeatable tests are ran on a physical storage device on an mbed board hooked up to be reseted at various intervals. This is the highest level integration test and is intended to catch anything else that we may have missed. (provided by @c1728p9).
https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_recovery/resilience_functional
@c1728p9 has also set up a long-running soak test using a set of repeatable tests that run indefinitely. This is intended to catch any issues that may crop up from the system being ran for a large length of time.
https://github.com/ARMmbed/mbed-littlefs-soaktest
In total, the tests take roughly 45 minutes to run on a single device with a single type of storage device.
How do I start using littlefs?
Switching from the FAT filesystem to littlefs is a easy as changing the class name from
FATFileSystemtoLittleFileSystem. If you're using storage-selector, littlefs is already available as theLITTLEoption.Here's a sed script:
When should I use littlefs vs the FAT filesystem?
At this point, the only remaining benefit to using the FAT filesystem is when the user expects to be able to plug in the storage medium to a PC. There is littlefs-fuse, which is a start at supporting the ability to mount littlefs on PCs, but it is currently limited to Linux.
What else needs to be done?
The biggest task remaining is to make sure that CI is able to run with the incoming tests. I'll be working with @studavekar to make sure the CI is set up with correct hardware.
A lower priority task is to get the incoming filesystem tests working on the FAT filesystem as well. @deepikabhavnani has put a lot of work into this so it should just be an effort of migrating her PRs on mbed-littlefs.
Additionally, there are several small patches in various places that are required to get everything working:
Allow a host test to reset the device under test htrun#174Fixed mutex assert in armcc fopen and related memory leak #5526integrate .travis.ymlcc @c1728p9, @deepikabhavnani, @studavekar, @sg-, @JanneKiiskila, @marcuschangarm, @MarceloSalazar