CubicWeb

During this period, we tried to fix issues, migrate some of our code base to latest version of CubicWeb. We also pursue some archeology tasks : merging or closing some merge requests from the old heads in CubicWeb's repository. We also release new ... [More] version of logilab-database, RQL and CubicWeb! CubicWeb 3.31, RQL 0.37 and logilab-database 1.18 are out! We released Cubicweb 3.31, RQL 0.37 and logilab-database 1.18 on may 4th. The biggest change is the addition of two new RQL options: NULLSFIRST and NULLSLAST. The NULLSFIRST and NULLSLAST options can be used to determine whether nulls appear before or after non-null values in the sort ordering. By default, null values sort as if larger than any non-null value; that is, NULLSFIRST is the default for DESC order, and NULLSLAST otherwise. Furthermore, a lot of issues have been fixed, and a few new features have been implemented. Here is an extract of the changelog: handle same_site cookies configuration in pyramid.ini rql: add support for order by NULLS LAST and NULLS FIRST improve default cubicweb skeleton dbcreate: don't ask confirmation to create schema in automatic hooks/notification: BREAKING CHANGE correctly initialize operation with event attribute RQLExpression: performance issue on RQLExpressions using EXISTS() BREAKING CHANGE: explicitly use EXISTS in RQLExpression for permissions fix some security issues One can find the full changelog on Cubicweb release page here. Adding python types to RQL In the past months, python types have progressively been included in RQL. This is a step forward to python types in CubicWeb itself. A lot of work has been done, and there is still a lot of work to do. Adding types to RQL help us to simplify the refactorings. As RQL's API is not clearly defined, we will run CubicWeb's tests on each merge request of RQL, to make sure nothing breaks. Future works More hackathon, more developpements! To have more time to develop CubicWeb, we will try to save a day each month to make a hackathon. Better handling of large databases In some CubicWeb projects, we have more and more data to manage; and databases are getting bigger and bigger. In the near future, we need to optimize the way CubicWeb deals with large databases. Table partition? More options to create indexes? We are also planning to have part of our databases checked by db experts to see what we can do. See you next month! [Less]

It has been quite a time since the last public news ; we will try in the letter to summarize what we did during february and march 2021. During this period, we tried to fix issues, migrate a lot of our code base to latest version of CubicWeb. We also ... [More] did some archeology ; we have been looking at all the old heads in CubicWeb's repository, and we have tried to rebase them on the latest public head. A lot of merge requests have been created and are still under review. CubicWeb 3.30 is out! We released Cubicweb 3.30 on march 16th. There are no “big” changes in this release. A lot of issues have been fixed, the documentation have been improved − a new tuto is coming ! − and a few new features have been implemented. Here is an extract of the changelog: it is now possible to use smtp authentification to send an email (#221) required variables can be read from the environment (#85) one can specify scripts attributes when using the add_js function (#210) GROUP_CONCAT was not returning the expected results when NULL values were encountered (#109) it is now possible not to drop table indexes when using the MassiveStore (#219) One can find the full changelog on Cubicweb release page here. Adding python types to RQL In the past months, python types have progressively been included in RQL. This is a step forward to python types in CubicWeb itself. A lot of work has been done, and there is still a lot of work to do. Adding types to RQL help us to simplify the refactorings. As RQL's API is not clearly defined, we will run CubicWeb's tests on each merge request of RQL, to make sure nothing breaks. Separate Back and Front For some time now, the trend has been towards a clear separation between front and back. We are adding more features to use React in the front. CwClientLibJS, a library allowing to use rql in the browser. CwClientLibJS is now in version 1.1.0 and is able to generate query on entity state (see !20). react-admin-cubicweb, new tools to generate admin pages. react-admin-cubicweb is still at an early stage but can display entity attribute and relation. It also allows edition as well. The version 0.3.0 has been released ! Release-new Release-new is one of our latest utility to release new versions of our cubes. Assuming that you are using conventional commits, you should really like this tool to release new version of your cubes. Release-new takes care to: update the version of the cube in __pkginfo__.py update the debian/control if there is one update the changelog create a new commit with theses changes tag the commit By default, it will try to guess if it is a major, minor or patch release (you can specify the release type if need be). The changelog will be updated and you will be able to edit it before the commit is done. The update of the changelog will be eased if you use conventionnal commit as your commit history will be dispatched in the appropriate sections. For instance, the last changelog of CubicWeb has been produced by this tool ! Docker images During our last hackathon, a team worked on our CubicWeb Docker images. The code to generate them has been rewritten to be simpler. During this refactoring, we took care to generate docker images for minor versions of CubicWeb as well. Therefore, on https://hub.docker.com/r/logilab/cubicweb/tags you can find docker images for major and minor versions of CubicWeb with different versions of python, to suit your needs the best we can :) See you next month! [Less]

New organisation For this new year, we decided to change how we organise the CubicWeb project at Logilab. We clarified how to define priorities and how to contribute. Rotating the coordinator of the month First of all, we pinpointed the need to have ... [More] a coordinator to organise meetings, issues and dispatch work between contributors. It was an evidence to define this role as a rotating responsability. Each month a coordinator is picked from the volontary participants. The previous coordinator and the new one have to write this monthly review together to communicate on what happened for the CubicWeb project during the month and to officially change the coordinator. New kanban boards To separate the on-going issues and the planned issues for the next versions, we divided the existing kanban in two. The first one defines the ongoing work and the states of each issue. The column "IMPORTANT" is used to track the important bug which need to be fixed in the version in development. The R&D board tracks the ideas and the future versions. This board contains an "IDEA" column, which is used as a backlog. Then we can move the "IDEA" issues into the dedicated column to plan the issue for a specific future version. The "Exploration" label is set on issues that are not asking to implement a feature, but to explore the problem and find a good implementation plan. Ideally these issues will lead to classic issues to be implemented. The CubicWeb Forge is dead, Heptapod is our new friend After months in transition, we decided to announce that the official site to develop CubicWeb is Logilab's Heptapod instance in replacement of the CubicWeb Forge. Heptapod is a friendly fork of GitLab that supports Mercurial repositories, allowing to use all the features of GitLab and all the features of Mercurial (draft changesets, automatic evolution, etc). All cubes were migrated to the new forge. CubicWeb Forge is still available, but will be made read-only and will be shutdown in the future. Weekly meeting Every tuesday afternoon (UTC+1), there is a weekly meeting. We discuss the issues in the kanban boards to track progress and plan the work. Feel free to join us in the CubicWeb matrix room. CubicWeb sprint To work on specific issues we have decided to organize regular sprints. The first one occured on January 26th and 27th 2021 with ten participants. It was a success as good work was done. Separate Back and Front The main subject which was studied during this sprint was the front/back separation, which is an "Exploration" issue planned for the CubicWeb v3.30. CWClientLibJS We discussed improving CWClientLibJS to help JavaScript developers write RQL queries into the front application. We defined a draft API for the QueryBuilder class which will be used to write RQL queries into JavaScript codebase. This API will allow to write complex RQL queries easily, such as with optionnal parameters. And the YAMS schema will be used to add helpers to get entity from eid and to organize the RQL query response depending on the entity type attributes. The next steps will be to test this API into a project and study how to handle authentication. CWElements Another subject is how to generate frontend forms from the YAMS schema, such as the current CubicWeb.web implementation but all with the React framework. The result is implemented into the CWElements project: - handle relations - allow update queries. The ongoing issues for CWElements is to test this form into a project and define how to establish a graphical identity by specifying CSS files and/or CSS framework (such as Bootstrap, Material-UI and so on) Cleaning up the CubicWeb code repository We corrected some bugs listed below, but more importantly we transformed all the branches from the old forge into merge requests in the new forge. Each merge request will thus be properly tracked and reviewed. If you have a patch for CubicWeb, now is a good time to send a merge request. The merge requests will benefit from the automatimated tests. Also, all the discussion will be properly stored. Important issues closed and merged [v3.30] [docker] allow to read REQUIRED variables from environments #85 [v3.30] remove statsd integration? #39 [v3.30] remove web.cors in favor of wsgicors with pyramid #192 [v3.30] RQL TODAY in sqlite is not equivalent to RQL TODAY in postgresql #109 [v3.30] Include pyramid as direct deps in cubicweb by default #191 [v3.26] Box view of a CWEtype is broken #74 [v3.26 & v3.30] No translation in pviews #87 Documentation Since we started working on Cubicweb 3.29, we have been reworking the documentation step-by-step. A new landing page was created. A new tutorial is being written. It focus on: Data import React to display a custom page (with RQL information) content-negociation to retrieve information in RDF format The fictionnal usecase is a website publishing a list of museum (imported from an existing source). The final result is compiled in the tuto cube. If you have any comment on the documentation, create an issue or come in the matrix room. [Less]

Hello CubicWeb community, It is with pleasure (and some delay) that we are proud to annonce the release of CubicWeb 3.28. The big highlights of this release are: CubicWeb handle content negociation. You can have get entity as RDF when requested in ... [More] the Accept HTTP Headers (see this commit for instance) CubicWeb has a new dynamic database connection pooler, which replaces the old static one. (see this commit for instance). RQL resultsets now store the variables names used in the RQL Select queries. It should ease the use of rsets and will allow to build better tools (see this commit) CubicWeb now requires python 3.6 as a mimimum. A big upgrade in our CI workflow has been done, both for tests and documentation. The development of CubicWeb has moved to Logilab's heptapod forge. To get more details about what has been added, modified or removed, you can have a look to the complete changelog published in Cubicweb's documentation. CubicWeb 3.28 has been published : on pipy.org as a python package. on our debian repositories as a Debian package. on hub.docker.com as a docker image. The tag latest now targets the 3.28 release. CubicWeb 3.29 is now on it way. We will have tomorrow (July 3rd 2020) afternoon a v-sprint (friday-sprint) to work on the documentation of CubicWeb and its satelites. See you there ! [Less]

Hi everyone, Here is the weekly report of last week meeting with some delay... Kanban status You can check the milestone here Build the new cubicweb image for all the intranet apps on the public head #9 Add py{27,3}-from-forge to clients project to ... [More] ensure we don't break everything when releasing #37 MR waiting on francearchive, Laurent is going to talk to Katia about it. Also Simon says that everything is fine with 3.28rc1 on data.bnf all internal apps of logilab runs with 3.28rc1 and there is not bugs to signal Todo Setup a demo with a SPARQL API this work has been started few years ago but today clients are interested in this feature option to create an RQL-SPARQL bridge would be too long work in progress to compare RQL and SPARQL expressiveness, not done yet current lead: using rdflib-sqlalchemy by adding tables next to existing Cubicweb tables and creating a prototype of an OWL to YAMS converter Rollback class_deprecated modifications on logilab-common Continue typing other libs such as: cubicweb (complex) rql logilab-mtconverter Patrick will probably start with this before continuing on RQL logilab-constrain logilab-database Identify "good first issue" to ease contributing Think about the documentation structure and what we want to write (for the next release) reduce technical debt spread documentation improvements among several sprints Current work working on fixes for YAMS tip for CW reducing the load on the CI by removing some useless tests when triggered from other CI adding needed commits on https://github.com/logilab/yapps to update it for python3 compatibility See you very soon for the next report ! [Less]

Hi everyone, We've just published the RC1 for CubicWeb https://pypi.org/project/cubicweb/3.28.0rc1/ and a new version 1.7.0 for logilab-common https://pypi.org/project/logilab-common/1.7.0/ Our current focus is finishing the last details for the ... [More] release. Milestone update the changelog is now part of the documentation of logilab-common to make sure it is visible test our clients project against our latest version on our repository to ensure we don't break everything when making a release allow 2 randomly breaking tests to fail (those aren't part of the code we are currently working on) Current roadmap 3.28, the version we are finishing right now, the current changelog is available here https://forge.extranet.logilab.fr/cubicweb/cubicweb/blob/branch/default/doc/changes/3.28.rst 3.29 on which we'll focus on writing documentation 4.0 in which we'll introduce the usage of semver for CubicWeb regarding its dependencies and release YAMS refactored version Semver One of our focus right now is to make stable releases of our core projects that won't break all the things ™ and we've made a lot of improvement in our testing suit to ensure that we test everything against our latest modifications before a release is made. Another problem we have right now is that CW only depends on a minimum version number for its dependencies, this mean that if we want to make a new release for one of the dependencies that will have some breaking code this introduce the risk of breaking all new CW installations. To solve this situation we have decided to implement semantic versioning and only introduction breaking changes in major releases and in addition to only depends on one specific major release at the time in CW dependencies. This way, when we need to make a new release with breaking changes, this will be a major release and we won't break all new CW installations. We have planned to start implementing this strategy starting CW version 4.0 Various updates a lot of fixes have been pushed on YAMS and CubicWeb to make CubicWeb compatible with the latest modifications in YAMS See you next week! [Less]

Hi everyrone, Version 3.28-rc1 is on its way! First, let's have a look to the issue board state. Milestone update Introduced types #10 logilab.common.deprecation has been typed (see hackathon report below): done Add tests for the content ... [More] negociation !20: MR about to be accepted Update logilab-common changelogs #43 : done Add automatic doc re-build to the CubicWeb CI #8 : done Todo Review and accept MR !20 Release logilab-common and cubicweb 3.28-rc1 Semver discussions Right now, dependencies are only specifying a minimal version. So if we introduce a breaking change in a new version, apps might break too. We plan to follow semver convention to prevent this from happening. We also discussed the idea of aligning version between compatible tools, so every major version would work with the same major version of other tools/dependencies. This idea will be introduced in 3.29 documentation, but will probably start with the release of Cubicweb version 4. Hackathon Last Friday we did an internal hackathon at Logilab and Laurent, Noé and I spent time working on Cubicweb. We mainly: wrote changelogs for: logilab-common cubicweb tried to add a Merge Request template on Cubicweb doesn't work on Heptapod actually, we will ask Octobus to have a look (see #46) added annotation types on logilab.common.deprecated improved tox.ini and added a gitlag-ci.yaml file in cube skeleton That's all! You should receive and email soon about the rc1 release. Thanks for reading, Henri [Less]

Hi, Welcome back to another weekly report! Today, the following topics have been discussed. Broken tests situation, follow up Migrating CubicWeb to Heptapod and modifications in dependencies resulted in broken tests as it was presented last week. ... [More] Work has been done on Friday afternoon thanks to Simon and Laurent, but it's not fixed yet. Tox is now happy but we still have a bug on a test that succeeds locally but not when run by the CI job. We do have a lead which may concern firefox usage in headless mode. Jobs logs are available here. Milestone update Introduced types Types have been added in Yams, merge request about to be reviewed We choose to dissociate this issue from the actual release as it's more related to Yams and not mandatory to release a 3.28 version. Re-build ReadTheDoc for every release Done for most of the dependencies but not CubicWeb yet Two dependencies, mtconvert and constraint don't have doc nor tests, we think about removing them instead of creating and maintaining this code which is pretty old Move to semantic versionning we talked about improving dependencies requirements to ease the release process (not giving only one version) we think we should stick to semver, but we need to discuss it more (should we bump all major version to the same number for interoperable dependencies, etc...) As those questions need to be discussed, we choose to move this issue to the 3.29 milestone Add tests to content negociation Need to be done Check if ?vid=rdf is still working Done We did spot that requesting vid=rdf or using content negotiation would not return the same RDF. We should fix this to have a more consistent behavior. Will be added for the next release. Todo before releasing version 3.28 To sum up, before releasing the next CubicWeb version, we need to: Fix CI tests Add automatic doc re-build to the CubicWeb CI This should be done and released within the next two weeks. Side notes For the next release we should align CubicWeb with changes made in Yams Release early, release often See you next week! [Less]

Hi everyone, Yesterday we held a new CubicWeb meeting, and we would like to share ... [More] with you what was said during that meeting : some issues have been migrated from cubicweb.org forge to the heptapod forge. You can find them here. Only the issues which were in the cubicweb-dev-board have been migrated, the others have been considered too old to make their migration worthwhile ; new milestones have been created, and some issues have been affected to them ; if you want to help out with Cubicweb's development, you can have a look to the Issues Board, and pick one task in the “To Do” column (this task should be related to a milestone) ; CW tests are still failing on the forge, and it's also related to other packages that have been released. Fixing those tests is quite urgent now, therefore we suggest to fix them in a sprint this Friday afternoon. Feel free to join ! On main Cubicweb's dependencies (RQL, YAMS, logilab-common, etc), a heptapod job has been added to trigger CW's tests with the last version of the dependency in the forge and not only the last released version on pypi. This should help to release new versions of CW's dependencies with more confidence. (for now, it only triggers the job, a future version of heptapod should provide a better integration of multi-project pipelines) ; The documentations of CubicWeb's dependencies are now automatically published on readthedocs. The work is in progress for CubicWeb itself ; Next week, if the tests are successful, we will talk about a release candidate for 3.28. Stay tuned :) [Less]

Yesterday at Logilab we had a small meeting to discuss about a roadmap to Cubicweb 3.28 (and beyond), and we would like to report back to you from this meeting. Cubicweb 3.28 will mainly bring the implementation of content negotiation. It means that ... [More] Cubicweb will handle content negotiation and will be able to return RDF using Cubicweb's ontology when requested by a client. The 3.28 will have other features as well (like a new variables attributes to ResultSet that contains the name of the projected variables, etc). Those features will be detailed the release changelog. Before releasing this version, we would like to finish the migration to heptapod, to make sure that everything is ok. The remaining tasks are: fixing the CI (there is still some random failings, that need further investigation) migrate the jenkins job that pushes images to hub.docker.com on heptapod, to make everthing available from the forge. It will be explicit for everyone when a job is done, and what is its status. Beside of releasing Cubicweb 3.28, its ecosystem will also be updated: logilab-common, a new version will be released very soon, which brings a refactoring of the deprecation system, and annotations (coming from pyannotate) yams, a new version is coming. This version: brings type annotation (manually done, a carefully checked); removes a lot of abbreviation to make the code clearer; removes some magic related to a object which used to behave like a string; The goal of these two releases, is to have type annotations in the core libraries used by CubicWeb, and then to be able to bring type annotation into CubicWeb itself, in a future version. On those projects, some “modernisation” has been started too ; (fixing flake8 when needed, repaint the code black). This “modernisation” step is still on going on the different projects related to CubicWeb (and achieved for yams, and logilab-common). In the medium term, we would like to focus on the documentation of CubicWeb and its ecosystem. We do know that it's really hard for newcomers (and even ourself sometime) to understand how to start, what each module is doing etc. An automatic documentation has been released for some modules (see 1, 2 or 3 for instance). It would be nice to automatize the update of the documentation on readthedocs, update the old examples, and add new ones about the new feature we are adding (like content negotiation, pyramid predicates, etc). This could be done in team Friday's sprint or hackathon for instance. CubicWeb would also need some modernisation (running black ? and above all, make all files flake8 compilant…). Regarding CubicWeb development, all (or, at least a lot of) cubes and Cubicweb related projects moved from cubicweb.org's forge to our instance of heptapod (4 and 5). Some issues have been imported from cubicweb.org to heptapod. New issues should be opened on heptapod, and the review should also be done there. We hope that will ease the reappropriation of the code basis and stimulates new merge-requests :) To end this report, we like to emphasis that we will try to make a « remote Cubicweb meeting » each Tuesday at 2 pm. If you would like to participate to this meeting, it's with great pleasure (if you need the webconference URL, contact one of us, we will provide it to you). We also created a #Cubicweb channel on matrix.logilab.org ; feel free to ask for an invitation if you'd like to discuss Cubicweb related things with us. All the best, and… see you next Tuesday :) [Less]