|
30 October 2024
|
Major Goal Achieved, and Next Steps
At long last, the sorter has now reached an accuracy of greater than 90%! This is both for object type identification (bricks, plates, slopes, etc.), and for part number identification (2x4 brick, 1x8 plate, 2x2 tile, etc.). What this means is that a huge step has been taken towards deployment. The more accurate the sorting, the less human intervention is required (less labor) and more objects are processed. At 90%, inaccuracies become bearable.
As exciting as this is, further testing is required. It's very important that this system be robust and reliable prior to deployment and during regular sorting operations. I'll only know that if I test it to the breaking point. Knowing the limits of the sorter will be very important to knowing how to best use it. I also want to provide an informed, honest, and demonstrable assessment of the system to current and potential users.
In addition to object ID testing, finalizing the BOM and other documentation will continue, and should easily be finished before the testing is complete. This documentation will also contain a list of objects that the system will be able to identify. At first, it's a pretty short list (58 common part numbers in 6 categories), but this will increase with regular updates to the ID models. A full list of all part numbers will, of course, be available to users.
On the hardware side of things, my plan is to completely tear down the prototype system and then rebuild it according to the assembly and calibration instructions that will be included in the deployment package. This exercise will fill in any details that have been sofar missed, and hopefully highlight any complications to successful deployment.
|
|
8 September 2024
|
Assistant
Now is a good time to mention that I've been using claude.ai as an assistant to help me generate data management scripts when I need them. I'll also get some help from claude when it comes to finalizing the layout and 'feel' of the GUI.
Sometimes it's fun to play with claude. Here I've asked it to generate a sonnet based on some basic information about the sorter. After a couple iterations this was generated:
In times of old, a chaos reigned supreme,
Of objects mixed, a jumbled, tangled mess,
But now arrives a sorter like a dream,
With steel and code to conquer randomness.
Its robotic arms, precise and swift,
Guided by eyes that never tire or blink,
Through piles of clutter, they sift and lift,
While processors ceaselessly think.
Machine learning's might, a digital brain,
Discerns patterns human eyes might miss,
From size to shape, no detail in vain,
Each item finds its place with algorithmic bliss.
This marvel sorts at speeds beyond compare,
From smallest specks to bulkiest of loads,
A task that once would drive men to despair,
Now done in moments, as the machine codes.
Order from chaos, a sorted domain,
Where every object knows its rightful space,
An ancient problem solved, no longer pain,
By vision, learning, and robotic grace.
|
|
28 August 2024
|
Training and Deployment Update
There are currently three things holding up deployment. First, the sorter must operate autonomously, quickly, and accurately. I think I've dealt with most failure modes, so hopefully the autonomous portion is satisfactory. The whole idea is that except for periodically refilling the dispenser and emptying the bins, the sorter is completely "hands off". Calibrate, click on the 'Go' button, and walk away. It should be as easy to operate and unobtrusive as your dishwasher! Speed continues to improve, but of course there's an upper limit to that. I'm still running the system in a diagnostic mode so no speed tests have been done. The sorting accuracy continues to improve with more training data, but it hasn't met my minimum standard of at least 90% yet. Low accuracy means either manual intervention is required, or automated sorting takes longer, or both. Until the system can meet this standard, deployment will be on hold.
Second, the Bill of Materials (BoM) must be detailed and complete. As far as I'm able, I will include links to suppliers, prices, and descriptions. I continue to work on this while training is taking place.
Third, the assembly and operating instructions must be complete, simple, clear, and unambiguous. I've deliberately designed this system to be easy to assemble, operate, and maintain -- but there is still some very minimal DIY skills required. A basic understanding of camera alignment would also be useful. I'm working on this while training is taking place.
Bottom line -- collecting data for training, and generating and testing new prediction models is the primary holdup. Higher accuracy seems to be proportional to the amount of training data, so it's not a matter of if, just a matter of when. I'm pretty close to that first 90% threshold, so it hopefully won't be too much longer.
|
|
04 July 2024
|
Closer and Closer to Deployment
As you can read in my previous entry, my plans were to create a few more demo videos showing sorting by object type. This, however, has not happened yet. Upon a more careful examination of the data coming from my tests, it was obvious that one of the object types (slopes) were being confused with other object types at an unacceptable level. And, likewise, some object types were being confused as slopes -- also at an unacceptable level. Nothing is perfect, but in order to be a useful sorting assistant the system needs to correctly identify object types, colors, and sizes at least 90% of the time. The confusion it was having with the slopes made the system not as good as it should be. The solution -- more training data.
I came up with a way of automating the process of collecting new training data, and went to work getting that all set up and running. Now I've noticed that throughout this entire project -- as I raise the bar on quality, more and more "little things" become very important. "Show stoppers" I call them. I've encountered and dealt with two big ones over the past two months. One had to do with dealing with the inherent imperfections associated with alignment (which is part of the all-important calibration process). The other had to do with how objects are being grabbed. But now that beautiful solutions have been found for those problems, the system is more accurate and robust than ever before! For example, a missed grab means that the system has to attempt to grab the object at least one more time. That slows everything down -- a real bummer. But as a result of this recent work, the number of missed grabs has dropped from nearly 20% to about 2%!! Because grabs hardly ever need to be repeated, this improvement translates to saved time and/or more throughput.
Here are some images of grabbed objects:
I also set up a worksheet for people to run some numbers to see if an automated sorter is something useful or not, and if it meets an expected ROI (Return On Investment). You can check it out
here. I used claude.ai to assist me in the generation of this. An incredible experience in and of itself!
I've frozen further development of the system so that all my efforts can be focused on deployment. So now I'm full throttle on three things: collecting more training data, training new foundation models (including models that can sort by part number as well as part type), and generating the BOM (Bill Of Materials) and a User's Guide for individuals or companies that want one or more of these at their location. Demonstration videos will be produced as soon as the models can more accurately (to my satisfaction) determine object type and part number. There are many other things going on in terms of giving the system a nice polish, but I'm glad to be able to say that the sorter is almost ready for deployment. Deploying this sorter at multiple locations throughout the world is my goal. If you're interesting in having a sorter at your location - or maybe near you - sometime in the near future, please check out my opportunity page.
|
|
23 May 2024
|
Improving Color, Defining Piles, Lights, and more Demos
After I made the last demonstration video showing the system sorting by object type, I set to work on generating videos demonstrating sorting by type, color, size, or any combination of these. To do this, piles need to be defined. That is, for example, put small blue plates into pile 1, large red slopes into pile 2, all technic objects into pile 3, etc. Creating these definitions is a necessary step in the calibration process to get the system ready for a sorting session, but I hadn't completely developed an easy way to do this on the user interface. To get demo videos out quickly, I could have manually defined piles, but I decided to step back and get it all working in the software (which had to be done anyhow, so why not now?).
Along the way to doing this, I discovered a new and much better way to do color measurement and matching. Since this was going to be a new addition to the sorting software, I developed a standalone version for development and testing. Once I got that working the way I wanted, I could then (more or less) just plug in to the sorting code.
This is about the 3rd or 4th iteration on color detection, so I anticipated that I was going to fall deep into the color rabbit hole. This time was no exception. Getting the standalone program working correctly was a can-o-worms, and then integrating it into the main sorting program was a bigger slimier can-o-worms -- but now that task is complete and the results are incredible. In addition, I created an easy way to define piles. Basically the user just needs to lay out some examples of objects in a particular way, and the system will determine the color, size, and object type (or part number) for each pile. Up to 15 piles can be user-defined, and there will always be one pile for 'unknown' objects.
On top of all that, I made a huge improvement to the lighting situation. Since the beginning of this project, I'd been using a 1300 lumen rechargeable LED worklight. Rechargeable also means it's a DC power source. Several months ago I decided I wanted to look at brighter lighting, so I switched to a 3000 lumen worklight. Taking the first image with my cameras reminded me of a cold hard fact: if the light is flickering at 60Hz (since this light has an AC power source), and my camera is running at about 25Hz, there will be dark horizontal lines in my images. This makes the system useless, so I put the brighter light away. I was still looking for a brighter light, but I was forced to look at 'rechargeable' ones since I knew they didn't have the AC flicker. This felt very limiting and so I started to think about how I could use the brighter AC powered light. A solution finally came to me, and it's turned out very well! I like the idea of choice when it comes to lights. Now the user can use whatever they want -- either AC or DC lights, with a lumen spec that will be detailed in the installation and calibration instructions. The solution I came up with also increased the signal-to-noise of the images by more than a factor of 2. The quality of the outputs are defined by the quality of the inputs.
Demo videos will be coming soon!
|
|
4 April 2024
|
Demo Video
A demonstration video showing sorting by type is now on my YouTube channel. My commentary throughout!
|
|
3 April 2024
|
Sorting By Type
Different types of objects sorted into piles. Click on the image for full-res. Top row, left to right: misc technic (gears, connectors, etc.), technic bricks, slopes. Bottom row, left to right: plates, bricks.
The sorting isn't perfect, but it definitely good enough to be a valuable resource. This particular set sorted 139 objects with eight correctly identified missed grabs and 19 mis-identifications. That's an accuracy of 87.1%, which is similar to what I've been seeing with testing. Notice in the image below that the sorter has the hardest time with bricks and technic bricks, but has no trouble with misc technic or plates.
A demonstration video showing sorting by type is coming very soon!
|
|
1 April 2024
|
No Fooling!
I've recently been testing an object identification model to identify seven different object types: bricks, plates, slopes, technic bricks, additional technic components (like technic gears and connectors), missed grabs, and unknowns. These categories were deliberately selected to make a significant impact on reducing your bulk backlog. The notable omission at this stage is the 'tile' category, which is pending due to the current lack of comprehensive data for training purposes. The model currently has an accuracy of over 90%, proving to be a promising tool for your sorting process. Future developments will aim to include more object types and part numbers, as we strive to catch up with a backlog of over 70 years. The first major milestone will be when the model will accurately identify the 15 most common object types and the top 100 items.
I'll be posting a number of new demonstration videos soon showing the sorting process by color, size, type, and all combinations of those. Stay tuned!
|
|
25 March 2024
|
Breakthrough!
At long last, I can say with certainty that sorting by object type is a reality!
The floodgates can now be opened. Details soon...
|
|
2 October 2023
|
Update
Precipitated by an unexpected life change (I suddenly found myself unemployed), and because I realized that I had no competition even close in this particular market, I decided to take an unscheduled break for a few months. In addition, the location where I'm working is very warm and uncomfortable, especially for a robot. The unusually warm summer pretty much forced me to shut down anyhow.
Life has once again stabilized (whew!) and I'm gearing up to get back into it and make a final push to object identification, which I know very well is the 'holy grail' that everyone wants. The ML process is tedious and full of ups and downs, but I'm very very close to finishing this and the preliminary results look incredible. I've made some modifications to the hardware and software in order to support object ID, and I'll talk about those in a future blog post. I'm ditching my initial idea of offering a sorting service. That didn't go as I hoped it would. People aren't as enthusiastic as I expected mainly because there've been too many promises made by others in the past, and everyone is very very skeptical. I did have some great conversations with a couple of companies who showed an interest, but they're in a holding pattern to see what my next move is.
That next move is right around the corner.
|
|
9 Feb 2023
|
Demo Video
Just dropped a new video showing the sorter working with a different type of object than what was in previous demonstration videos. There are practically no limits to what this system can sort. Anyone who's really paying attention will also notice that object management has changed a little, too. Most of the improvements and changes being made aren't obvious except when you see how much better the system is behaving as time goes on.
|
|
30 Jan 2023
|
Defining Piles
Defining which object goes into which pile is up to the human user. Currently up to fifteen different piles can be defined. In the images below, sample objects of seven different colors have been placed in a calibrated pile designation grid to tell the system what color(s) goes into which pile. The software will automatically measure the color and assign it to the sorting pile specified. Many configurations are possible, including combining different colors into one pile.
| Pile | Color | Pile | Color |
| 1 | None | 9 | Black |
| 2 | Blue | 10 | None |
| 3 | None | 11 | None |
| 4 | None | 12 | White |
| 5 | None | 13 | None |
| 6 | Yellow | 14 | Red |
| 7 | Dark Gray | 15 | None |
| 8 | Light Gray |
|
|
23 Jan 2023
|
Demo Video
Just dropped a new video showing the functionality and capabilities of the sorter.
|
|
22 Jan 2023
|
Yes, but ...
You: Your sorter takes 15 minutes to do something I can do in 5 minutes
Me: Yes, but you don't have to!
|
|
21 Jan 2023
|
Selecting a Color Palette
Users can select how many different sorting bins to have, and what colors go into those bins. Specification is done by either manually entering the information into a metadata file, or have the system automatically measure the color and assign it to a bin. Currently, up to 768 different colors are available to choose from.
Here are the colors I use in the demonstration videos, arranged into seven sorting bins:
|
|
16 Jan 2023
|
A new demonstration video: Robots are Persistent
|
|
11 Jan 2023
|
Sort By Size
The sorter will now sort by object size, as well as by color. Currently the software is configured to allow the human user to specify up to three different sizes for each pile: small - up to about 1.3 square centimeters, medium - 1.3 sq cm to 10 sq cm, and large - 10 sq cm and larger. There can actually be as many colors and sizes as the user wants as long as there's physical space for the sorting bins and the robotic arm can reach them. For now, I think three different sizes is adequate for this working prototype system.
This is a pretty significant addition of capabilities -- each sorting bin has one color and up to three sizes associated with it. For example -- one bin can be for small red objects, and another bin for medium and large red objects; another bin can be for all sizes of dark bluish gray objects, and there can be a bin for large yellow objects. There's always a bin for 'unknown' objects also. Lots of combinations are possible, giving the user the flexibility needed to sort in many different ways.
|
|
3 Jan 2023
|
Launch Day!
Today Third Arm Robotics launches it's first product -- the Standard Sorter v1.0!
"I said 'lunch', not 'launch'!!!" -- one of my earliest and favorite puns from a typically bizarre Sid & Mary Kroft TV show called 'Far Out Space Nuts' from 1975 (co-starring Bob Denver -- it only lasted one season). I've been saying this to myself all day!!
|
|
™
