Forums : Technical Issue Help

In
https://www.ohloh.net/p/ros-pkg/enlistments?page=9

some checkout failed, but have long since been back up. Why does Ohloh not retry after 24h or so?

https://www.ohloh.net/p/ROSorg
Stats here are similarly not updated.

Can the manager at least get mail notifications when this happens?

Thibault Kruse about 12 years ago

Any update?

Thibault Kruse almost 12 years ago

Thibault,

It's on the top of my do-and-see list for today. I'll get back to you soon, good or bad.

Thanks!

ssnow-blackduck almost 12 years ago

Thibault,

There's 29 failed jobs, most mercurials which failed due to abort: error: EOF occurred in violation of protocol. It's one that we haven't been handling automatically; probably because it's seen so seldom. We only show 65 failures of this type currently and nearly half are from this one project. I did find two interesting things, though: one enlistment from this project was disabled after the failure occurred. Also, two enlistments from a related project (Willow Garage ROS Packages) were among the failed jobs. I'll also take a look at that project as well.

In the meantime, I'm on the job...

Thanks!

ssnow-blackduck almost 12 years ago

cool, thanks. Some repositories have been deleted since then (we talk about 10 months ago), wemigrated alot of repos to github. So I am trying to fix the ohloh project by removing now invalid projects and adding the new locations. Sorry, in the process I also removed and readded 2 projects, even if the help says I shouldn't.

Thibault Kruse almost 12 years ago

Hi, so I now see those repos that still fail,several of which have by now been taken down. Should I remove them or do you want me to wait so you can debug. Because as an example, on this page:
https://www.ohloh.net/p/ros-pkg/enlistments?page=8
Enlistment https://kforge.ros.org/rososx/homebrew
fails in Ohloh, while hg clone works fine for me.

Thibault Kruse almost 12 years ago

Thibault,

No worries there. The warning is more to make it clear that we remember history on a particular enlistment even after a delete. Delete is less permanent than it seems and is the equivalent of an undo to the enlistment in the edits menu and can always be redone. The only secret to it is to change the transport from https: to http: (or svn: or git: etc.) which we consider a different enlistment altogether. Sometimes these alternate transports or port numbers aren't available, though.

Thanks!

ssnow-blackduck almost 12 years ago

Thibault,

I would wait until the auto reschedule does its magic which often takes hours or more to start. Just checked and all the 65 failed jobs are no longer visible which implies they are either failed for another reason or finished OK or still running. I'll determine which right now and report back. There was at least one stalled job that had a problem for a different reason and I need to look at that one now too.

Get back to you soonest...

Thanks!

ssnow-blackduck almost 12 years ago

Thibault,

Four of the rescheduled jobs failed due to HTTP Error 404: NOT FOUND. They are /common/nodeletcore, /common/pluginlib, /robotmodel/urdfdom and /rososx/homebrew. These all need to be deleted and replaced with equivalent enlistments from elsewhere if available.

There are still three rescheduled jobs waiting to run so the jury is not in on them yet.

Thanks!

ssnow-blackduck almost 12 years ago

I can see that. I removed the failed that should fail with 404, and one of the pending one that should also fail. However,

https://kforge.ros.org/rososx/homebrew

should work. I can clone that one right now, so it should not give 404. Maybe it gives a different error. This repo is not crucial (contains 1 file), I could remove it, but the cause for this failure might be relevant to prevent other failures. Can you find out the exact error for this one?

Thibault Kruse almost 12 years ago

Also, this project suffers the same problem:
https://www.ohloh.net/p/wg-ros-pkg/enlistments
It says: https://kforge.ros.org/pr2apps/pr2teleopapphg Mercurial (Failed 8 months ago.)
but this repo is up now.

Maybe there was some error 8 months ago, but since then nonof the other enlistsments have been updated neither. Is this behavior of Ohloh going to be changed anytime? Because for our project, repos will continue to migrate, and sometimes servers might be down. It is a bit annoying if we have to go to the help forum each time to ask to restart the updates, for both us and you.

Thibault Kruse almost 12 years ago

Thibault,

Will do re: homebrew.

Will take a look at wg-ros-pkg and deal with any failures. If a failure is common, then it is a good idea to have those monitored. The once-in-a-blue-moon ones will always need manual intervention. There's always a gray area between the extremes. I will say my job has become much easier since we have had auto-retry implemented.

Not much we will be able to do to track the migrations until someone comes up with a bullet-proof way of flagging the changes for automated implementation. That would be welcome.

Thanks!

ssnow-blackduck almost 12 years ago

Sotoday I still see several jobs waiting in queue, I suspect that there is a problem with those. E.g.:

https://www.ohloh.net/p/ros-pkg/enlistments?page=6

and

https://www.ohloh.net/p/wg-ros-pkg/enlistments?page=6

I cannot find documentation on how auto-retry exists for ohloh enlistsments, nor how an ohloh project reacts to failures. E.g. does the manager of the project https://www.ohloh.net/p/wg-ros-pkg get a mail notifying of the failure? Oh, I see it currently has no manager. Hm, maybe it would help to display that there has been a failure that could not be resolved on the main project page, such that random users can see it and contact maintainers.

I'll try to get someone be manager for those ROS projects.

Thibault Kruse almost 12 years ago

Thibault,

Auto-retry is an administrative function and isn't documented publicly. Most failures can be determined on the enlistments pages, though not specifically. So far we haven't made more detail available since the ability to do anything with that information isn't available either.

There are pending jobs which are scheduled and have not yet run and it's longer than I'm comfortable with as well, but I don't have deeper control of the jobs beyond scheduling them. I will request someone up the chain have a look, though.

Thanks!

ssnow-blackduck almost 12 years ago

All projects seem to have updated now, thanks. will check every now and then.

Thibault Kruse almost 12 years ago

Thibault,

Looks right to me as well.

Please tell me here if the projects stumble again.

Thanks!

ssnow-blackduck almost 12 years ago

So, again there was a failure:

https://www.ohloh.net/p/ROSorg/enlistments?page=2
git://github.com/ros/robot_model.git master Git (Failed 7 days ago.)

The repo checks out okay now.

This is a bit vexing, it is not really viable for us to check ohloh every few weeks, is there a plan to imrpove the general recovery of projects on ohloh after failures?

Thibault Kruse almost 12 years ago

Same here:
https://www.ohloh.net/p/ros-pkg/enlistments?page=5
https://github.com/ros-perception/perception_pcl.git master Git (Failed 7 days ago.)

I don't quite get it. Does this mean every time github or bitbucket hickups, all ohloh.net projects need to be manually reviewed and restarted?

Thibault Kruse almost 12 years ago

Thibault,

I'll take a look and report back today (though not right away...)

A lot depends on the nature of the specific failure. I'll be glad to e-mail you the exact error message in each case. We do have a good number of specific errors in auto-reschedule and that makes my life easier overall but some errors aren't so easy to automate and we just keep coming up with new failure modes (flawed humans do that... not me, of course!)

Thanks!

ssnow-blackduck almost 12 years ago

Thibault,

In the case of both projects, the problem is the same: Investigation shows there is no longer a master branch.

Our default for git repositories is to use the master branch. Should that be missing, a new enlistment needs to be entered where the desired branch to be monitored is entered under the Enter an optional Git branch name blank. Please create new enlistments in each case with your preferred branch and also please delete the old enlistments.

Please let me know if you have any questions.

Thanks!

ssnow-blackduck almost 12 years ago

I see, thanks. Fixed both instances, we move away from master branch usage. Would be niceif ohloh could one day pick up the default branch from github, not sure whether the API allows it (basically if the user leaves branch field empty, git clone would get the correct version9.

Else it might be nice to have github integration showing the ohloh build status on the github page.

Thibault Kruse almost 12 years ago

In both projects the enlistments have changed, but since 24h remained in the enqueued state.

Thibault Kruse almost 12 years ago

Thibault,

We are running behind and it may take a day or two to catch back up so all pending work has been completed. I'll keep an eye on it and let you know (probably on Monday) when I see that the project has run through the steps needed.

Remind me if I forget or let me know if you have any questions.

Thanks!

ssnow-blackduck almost 12 years ago

Thibault,

Initial fetch finished OK (73 total commits). A full update started immediately and also finished OK. Analysis is now at March 30, 2013 on code collected on March 30, 2013 and last commit is at March 29, 2013.

Please be sure you remind me here if the project stalls again.

Thanks!

ssnow-blackduck almost 12 years ago