CoW Backlog

(Maintained by Hamish.)

1. Roadmap (version 1.0)

1.1. Beta 2 (RFC from gate-users)

• fix the sandbox corruption problem [IR]
  • implement checkCancelled() in the SVN event handler; on shutdowns and on edits do cancellation of up jobs (and block until the current job completes)
  • better logging of the svn up jobs, with an obvious way to grep problems out of the logfiles (failed updates, hanging updates)
  • log shutdowns
  • operations on the staging area should use a diff (temporary?) sandbox object
  • for reference, the option we discussed was
    • all svn ups (periodic and as a result of edits) should go into a single priority queue
    • server shutdown should clear the queue and then block on the completion of the current update job

• load testing of videos and testing against g8rs.net [MG]
• jape yamalyser [HC, NA]
  • fix issues identified in cow/doc/*.yam [NA]
  • do the TODO in PageService.saveEdit [HC]
• webflows [IR]
  • add delete flow (operating on the selected files in directory view)
• g8rs.net [HC, IR]
  • fix the doc build which currently runs out of RAM on YamFormatter.java (use the grails doc target instead of groovydoc, and add more RAM / the normal classpath)

• misc bugs / minor features [all]
  • security stuff [IR]
    • the g:paginate call in jsecUserRoleMappings/listByRole.gsp needs an id param adding
  • when dependencies regeneration is triggered for a file containing includes, the regeneration seems to be run with includes turned off (e.g. edit the download page on g8rs.net, then look at /narrative.html - this now has the includes not processed). [AR]
  • directory listings should not include directories for which the user doesn't have read permission

• fixing the ge1 rsync probs
• fixing the selenium tests under hudson: seems worth trying to rewrite the tests that are unreliable, with more pauses and/or a diff set of selenium actions to do roughly the same things
• logging
  • turn it down a bit
  • use a fifo into a log rotator for the stdout log
  • add timestamps (though not in seconds!)
• robots.txt

1.2. 1.0

• merge in stuff from gate-server/html that has been added since the branch to gau-main-tree (e.g. the projects dir has had new stuff...)
• upload and newpage flows [IR]
  • upload of an empty file fails without a message
  • uploads and newpage go directly into the sandbox - they should go via staging, at least in server mode.
  • restrict file uploads so users can't upload raw HTML in a location they wouldn't be allowed to view it.
• rationalise the tabbing orders
• strangely, google search over the GMANE list archive returns less hits than the GMANE search (try e.g. "wyner"). perhaps add a GMANE search YAM plugin like the google one? see the GMANE search pages' "short url" link for what the request should look like
• put something on gatecloud.net
• double the amount of RAM for run-prod to increase the headroom (not done earlier to try and make sure it is headroom) and add more file handles while we're at it

2. Future Tasks

• edits of non-yam html delete META tags... options: fix the implementation of putBody in FileUtils and go back a non-full-page fck edit; upgrade to more recent fckeditor; try YUI rich editor. while we're at it try to get rid of PageService.FckEscapesFix
• HtmlTranslator excludes e.g. /g8/... from being a candidate missing page; the list of patterns should be a config option in cowrc.groovy that gets pushed into a YamFile static at bootstrap
• svn up operations should trap the file names that are updated and push them to the solr reindexing queue
• quartz jobs and perhaps also webflows should log their start/end times, and put something characteristic in all their log messages; then we'll be able to grep individual jobs/flows out of the log
• what happens to footnotes (or twitters) in included files? [HC]
• g8rs.net
  • get cron to publish security-related log reports somewhere (in the wiki space?) [IR]
• at bootstrap allow raw html in all of the help area (which must be readonly for non-administrators)
• put some navigation in the help area
• is _pageBody.gsp actually in use?
• can we replace helpdocs.zip with a zip of the help svnrep?
• email confirmation plugin
• add tooltips, e.g.: <g:javascript> new Tip( 'newpageName', '<g:message code="cow.new.page.title"/>', {title : '${message(code: 'cow.new.page.tooltip').encodeAsJavaScript()}'}); </g:javascript>
• if you put a link in an existing page to a new (non-existant) page in a subdirectory, you should get the same error message that you get if you try to do this via the new page form, but this doesn't work
• create a new wiki area with a non-existant directory as the target sandbox; the exception that gets thrown by SandboxManager is reported (though not nicely), but the wiki area still gets created. need to trap the exception, report it (and log it) and then abort the area creation. this probably also kills bootstrap when the SbManager call is being made during startup
• navigation menus: to support tablets (e.g. Valy's Nokia) a click on the area of a submenu parent (i.e. the menu item that has a submenu) as opposed to the link on the parent should open the submenu, not the parent (alternatively the arrow area should open the submenu)... difficult: perhaps buy valy a new phone?! or WAP-style tailoring of CSS to different devices, and while we're at it add skins?!

• add exclusions for / and /help to HtmlTranslator so that links to these areas are not considered to be missing pages [HC]
• add %center(...) to YAM
• team wiki ... [MG, HC]
• move cowrc.groovy inside cowrc.d? One less thing to reconfigure when you want to run more than one CoW on the same machine.
• fix conflict between Quartz versions [has been done during 1.1.1 upgrade?]
  • Grails provides Quartz 1.6.x
  • the Quartz plugin uses 1.5.2
  • these are incompatiable
    • currently fixed by placing 1.6.1 in the quartz plugin directory
    • THIS IS A NASTY HACK!
• Set the session timeout. See http://www.nabble.com/Jsecurity-setting-the-session-time-out-td20521251.html and any changes to plugin since it was written
• add ability to do syntax highlighting on code listings, with something like %code(...) using http://code.google.com/p/google-code-prettify/
• regeneration
  • setContext on the YamFile
  • use the PageService edit-related methods for regeneration (have a fake user and change PS to allow parameterisation of this)
  • if there's a conflict at commit time, delete and do nothing (cancel this regen; a real person did it...)
  • change PS to delete dependent HTML files before doing update; this will solve potential conflicts in HTML when YAM changes, for example
  • turn quartz job back on and check for conflicts problems

• grails 1.2
  • figure out how to get e.g. the gau site plugin to reference plugin files from the layout without the full path? (cf. the images links in cowguest)
• [CoW] dependencies
  • at the moment, dependencies regeneration runs in the foreground. But should it be a background job? background would be good e.g. overnight, but foreground might be better usually, as it is always run by an admin, who perhaps wants to know when it goes wrong? think about it.
• [CoW] paths
  • create a new install, login as admin:
    • gives: "About to create a new page (or directory) named cow... ok?"
    • but going in via admin browse wikis works ok
    • or, it sometimes gives a directory listing instead of the existing index.html
• [CoW] wiki functions
  • PS refactoring, and gate.util.Yammer
    • PageService could be more concise and clearer if it had an object model for the various types of page and the files on disk (gate.cow.Page, YamPage/TxtPage/BareHtmlPage/Directory extends Page, with overwrite/toStaging/fromStaging methods). PS.analyse would then put the appropriate type into the page model

• [YAM] a snippets plugin? for example, we might want to add a sourceforge mail search plugin... but this shares a lot with teh google plugin; shall we just go on adding more and more plugins to do small jobs like this, or could we allow CoW administrators to create snippets somewhere and then have a general plugin (like the Grails blurb plugin)? disadvantage: ties YAM to CoW, which is not currently the case...
  • <FORM method="GET" action="">http://sourceforge.net/search/">
  • Search <a href="mail/index.html">the mailing list</a>:
  • <input type="hidden" name="type_of_search" value="mlists">
  • <input type="hidden" name="forum_id" value="47765">
  • <input type="hidden" name="group_id" value="143829">
  • <input type="text" cols="30" name="words">
  • <input type="submit" name="Submit" value="Submit">
  • </FORM>

• [CoW] misc
  • autosuggest LinkedOpenData references during authoring?
  • using cow:pageLink etc.: this patch makes the pageLink taglib use the cow:namespace, but seems to break the normalExit state in newpageFlow; no idea why; perhaps try later when at grails HEAD again, and send bug report. in the meantime going with g:page...
  • docs stuff on tropical tolerance to cow docs, update sf.net http://tropicallytolerant.blogspot.com/ http://www.spectrum.ieee.org/print/1699
  • RSS: try the feeds plugin
  • add latex import and support lyx
  • exploit gant latex build for cow latex output
    • includeTool << gate.tools.LaTeX
  • look at S5 for slides output
    • http://www.g2s5.com/presentation/s5express http://www.odelia-technologies.com/g2s5
  • bibtex support via http://jabref.sourceforge.net/ ?
  • alternative approach to exception handling: http://blog.bruary.net/2008/03/grails-custom-exception-handling.html
  • round trip with freemind: http://freemind.sourceforge.net/
  • ability to link between areas, and link to help from doc/main-index-template.yam, for example
• [CoW] semantics integration
  • clone/questio
    • start from olaf's grails-gate-plugin code
    • make questio a grails plugin
    • (maybe) allow yam language to pull in configured predicates
    • make url mappings accept plugin controllers
  • %questio and %sparql (or %owl-nl and %owl-sparql?) in YAM; translation puts AJAX code that pings OWLService <HC, AR>
  • JS code to embed in cow (gatewiki.sf.net) pages that
    • contains a SERQL (or other) query
    • registers an on-load event
    • when the page is viewed in a browser, send the query to the Grails service and renders the result (inserts it into the DOM)
• [CoW] PageController <HC>
  • possible optimisations of the PageService check-out-to-staging code:
    • use the SVNRep API, see: http://www.nabble.com/Re%3A-check-out-a-single-file--p15719277.html
• [CoW] pre-publication area
  • /trunk/ and /branches/ and pre-publication edits and so on
    • create a single "editing" branch. or a new branch for every edit? every user?
    • a new branch can just be a new Wiki
    • new roles: production-editor vs. branch-editor? (former can edit live site)

• [CoW] security, authorisation and authentication
  • should the user's edit profile form require a password entry?
  • when user details are edited via the edit profile form they do not change in the interface until next login. Things like username on the interface etc come from org.jsecurity.subject.Subject.getPrincipal(). This doesn't change until the next login. We can't automatically log out and back in, as we do not know the password, only the hash. we need to somehow force the principal to change. Maybe you can set it? Maybe a question for the list / support?
  • Logging on SecurityFilter: have never managed to get logging working on this. See mailing list question http://grails.markmail.org/message/wqf7m7bymq6tx6f7?q=grails%2Eapp%2Efilter and the refernced JIRA, http://jira.codehaus.org/browse/GRAILS-3155
  • on the mailing list: http://tinyurl.com/b9j4xh The principal tag in the JSecurity plugin isn't HTML encoded and therefore vulnerable to XSS attacks. A patch would be to call encodeAsHTML on the result in the JsecTagLib. This has been done in our own code base, the cow:principal tag, and the question asked again of the mailing list: check there again. Filed a jira: See jira http://jira.codehaus.org/browse/GRAILSPLUGINS-657 Once it has been fixed in the plugin, we can replace cow:principal with jsec:principal
  • protect JsecActionList that are set in configuration from deletion, and warn about deletion of other actions sets if they are in use. This can be done in the same way as for JsecControllerName and JsecRole
  • Directories in the permission model are being changed to be a regular expressions (one for included directories and one for excluded directories). This could be changed to (a) allow a list of regular expressions on each permission and (b) use the easier ant syntax, rather than java regexp syntax.
• [GATE.ac.uk] "what we do page" <DD,VT> (or: MIMIR search):
  • A JSP that displays Di/Wim's taxonomy and allows users to select one or more concepts
  • When they hit "search" these concepts then become a keyword search against publications.dcs and the results are displayed (list of publications relevant to the set of concepts)
  • http://www.google.com/coop/cse/overview for site search?

• [YAM] Yammering <HC>
  • yam lists in table cells bug (or maybe just an irritant that needs documenting)
    • as for example in sam/doc/sheffield-visit-10-08/future.yam if the first item in a table cell is a list, it has to be preceded by a linebreak, otherwise it causes syntax errors
    • note that the syntax errors can be quite wierd: they include the list item problem, but also "EOF at ..." errors that aren't obviously related to the list (though they go away if you fix the list)
    • it would seem possible to get pushBackListStart to put add a sep token into the stream before the token being pushed back, or similar...?
  • finish fixing latex generation
    • uncomment tex extension in YamTranslatorTest.testHtmlToYamNoIncludes and fix remaining errors
  • %clone
  • slidy (sale/talks/gslidy)
    • should be done like PDF and PDF dropped
    • yam layer
    • doc for team
  • Version 5 bugs:
    • the verbatim mode for HTML doesn't deal properly with angle brackets (which need to be escaped). this is due to html pre not being like latex verbatim. perhaps add another verbatim mode (%<< ?) or make chevrons TargetControls in the lexer?
      • is this now fixed with the html escaping code?
    • commas and citations within footnotes will prematurely end the footnote, and any markup within footnotes will not be interpreted (see also next point), though \, is a valid workaround
      • one option is to have a member predParser and a JAVACODE production that reads it in, counting brackets, then parses it and merges into the tree
      • perhaps a better one is to make a new token "%footnote(" and allow Text() inside it, and mandate escaping of brackets?
      • also, converter xsl written on the assumption that there is no markup in the footnote. may need to be rewritten if this changes (see footnotes in html generation above)
    • (related to previous) left round brackets in URLs cause truncation of the url; we need to allow escaping or quoting (perhaps the same mechanism for all predicates?)
    • includes get wrapped in paragraphs, which results in empty tags in the html. solution: make include different from Predicate and put it under Unit
    • when an image alt tag specifier is enclosed in quotation marks the generated tag puts the attribute quotes in the wrong place
    • spaces at the end of table rows (after the final |) gets discarded
    • if %contents is used multiple times, only the last one works
    • profiling: (yourkit profiler?)
      • check if 1MB docs and above work (double yam-huge)
      • check if parse errors in documents above around 500k (e.g. yam-huge.yam) cause out of memory errors
      • If context isn't set on yams, and generate done withiut links, conversion of the site takes a few seconds. With context set, it takes nearly 3 minutes. (not such a problem for conversion, but an issue elsewhere?)
    • because blockquotes (and things like bold, italic) are just treated as words, they can cause overlap problems (see yam-wierd line 89 and after for example); this would be hard to fix - the workaround is to e.g. make sure to separate the blockquote markup from other constructs that it may enclose
    • an empty URL is illegal: %()
    • all list items, including empty ones, must have a space after the initial "-" or "o". (this is not a bug in fact.)
    • verbatim mode terminates lists; verbatim within lists doesn't correctly place the pre tags in the output and doesn't escape HTML properly
    • when including a file it should be possible to have the title as a higher level heading than the other headings (or simply to exclude the title)
    • if the last thing in a table is a verbatim (%<...>%) item, that thing will not be verbatimised [need to check if this is in the list of known yam bugs below nad if not add it]
    • images in links don't work, e.g.: %(tmp/summer.html, %image(tmp/Screenshot.png) ), and predicate arguments are also not parsed [HC]
  • Version 5 wishlist:
    • something like latex's ref command
    • definition lists
    • appendices
    • captions for figures and tables

• svn replication links: http://www.darcs.net/DarcsWiki/Tailor http://www.wandisco.com/

• [CoW] security on searching
  • At the moment, the search controller is a special case in the SecurityFilters. If someone is trying to access it, then the SecurityFilters checks for permissions on the page controller for the read action (if a user can read a page, then they can search it)
  • We could generalise this special case by allowing permissions to be defined over sets of controllers, in the same way we do for actions:
  • CowPermission(Wiki 1, directory "", controller [page, search], actions [Read])

3. Completed Tasks

A list of completed tasks is here.

4. About this Document

This document contains the list of features, functions, technologies, enhancements and bug fixes that would ideally be made part of CoW. It represents a snapshot of the ever-changing requirements and plan for the system. Items at the top of the list should be detailed enough and fine-grained enough to be implemented. Lower down the list are lower priority features that may or may not be implemented. There are no bad ideas for new features, only low priority features.

This document links to

• user documentation: e.g. the GATEWiki user guide
• developer notes: our design notes

Format:

• [Task type] Task content. (Time estimate.) <People>

The time estimates and people assignment are optional (they're most appropriate for tasks that are near the top of the stack, i.e. will probably be implemented in the coming period).

For example:

• [Core] Remove warnings due to Java 5 Type safety. (2 days) <JS>

When a task is complete move it to the Done section, and link it to relevant user and design documentation.