Forums : Technical Issue Help
Dear Open Hub Users,
We’re excited to announce that we will be moving the Open Hub Forum to
https://community.blackduck.com/s/black-duck-open-hub.
Beginning immediately, users can head over,
register,
get technical help and discuss issue pertinent to the Open Hub. Registered users can also subscribe to Open Hub announcements here.
On May 1, 2020, we will be freezing https://www.openhub.net/forums and users will not be able to create new discussions. If you have any questions and concerns, please email us at
[email protected]
WebKit project has not been updated for more than a year
The WebKit project https://www.openhub.net/p/WebKit has not been updated since more than a year.
I find this completely incredible. This is a high profile project. How can you miss that this was not updated for such a long time?
Please update it. Thanks
Also it will be awesome if you could implement automatic alerts for projects not updating.
Ping?
Hi Carlos;
This is nasty. As we process the repository, the server runs into a commit that sucks up all the memory. The actual error is:
Cannot allocate memory - cd '/var/spool/clumps/000/000/786/704' && git cat-file -p ab509f8ec068bed511129f1e7dc772f8f1e0585e
Note that the /var/spool/clumps/....
part is where the repository is located on that particular server. We have 18 backend servers with over 27TB of storage to manage over 680,000 repositories. The issue is that when the process tries the git cat-file
for that particular commit, it runs out of memory.
We occasionally see this and will try to figure out the problem.
Please note the same type of error blocked the project last about a year ago, although I am not certain that it was with the exact same commit ID. Because of that error, deleted all the old jobs and artifacts and re-started the process, but got the same results.
There are some issues
- The WebKit developers have the bad habit of uploading in the repository all kinds of medi (images and even videos) for the layout tests.
- git is very bad at handling binary data.
So unless you have a server relatively powerful (4GB of RAM and 10GB of disk space free at least ) git is going to have a bad time handling this repository.
Fortunately, there is a solution for this that don't requires you to have better hardware:
- The official repository of WebKit is svn. git is a read-only mirror.
Just switch the repository for WebKit from git to svn:
svn checkout https://svn.webkit.org/repository/webkit/trunk WebKit
https://webkit.org/getting-the-code/#checking-out-with-subversion
Ok, seems that is not the reason.. I'm checking that commit and it looks very strange.
It looks like an empty commit where the file is the git log message rather than the file ??? I have no clue what this is, but it don't looks like a normal commit.
Indeed, running git cat-file -p on it gives the samething than the previous git show pasted output. It lacks any tree/parent/author info :\ ?
Looks like this hash is not a git commit but a git blob object:.
It actually seems to be the blob hash from commit 4384979b970cc62ed02245bdcdf05ae6da6ffabd which is from SVN commit 9
Why are your scripts running into this???
Hi Carlos;
Thanks for all the sleuthing. This is very helpful and we'll keep digging.
As for the server, it really should be able to handle this. We run with 18 crawlers each with 12GB of RAM with 1.5TB of disk, of which over 280GB are available (on the server that encountered this error). And these are the old systems.
The Import process steps through the commit history and imports the record of the commit into the database for all the processing that will follow. There may be an opportunity to be selective about what is being imported and we'll pursue that line of logic.
We updated the version of git that we're using and unfortunately continue to encounter the issue. I cleared out old jobs and tried this one again. This time it died on:
git cat-file -p 70761c199cf7a93e3a29e4605f979a3ce89d01ad
I suspect it has nothing to do with the repository or the version of git. I think the code that executes this is poorly structured and is trying to load too many massive arrays of Diff data for comparisons in to memory and simply needs to be re-written. I'm opening a ticket to address this.