Sunday, 2012-04-15

GitHub146[mantisbt] davidhicks force-pushed next from 38b904b to 306956d: http://git.io/t1IA8Q01:23
GitHub146[mantisbt/next] Move config files to application/config directory.  This will be out of the web tree once restructuring is complete. - Daryn Warriner01:23
GitHub146[mantisbt/next] Move core files to application/core.  The .htaccess file will not be needed as this will not be reachable - Daryn Warriner01:23
GitHub146[mantisbt/next] Move Soap api to application/services/soap. - Daryn Warriner01:23
*** Joins: Paul_46 (~IceChat09@cpc1-enfi15-2-0-cust580.hari.cable.virginmedia.com)03:52
dhx1Paul_46: :)04:21
GitHub120[mantisbt] davidhicks pushed 1 new commit to next: http://git.io/P5rH5w04:30
GitHub120[mantisbt/next] Replace plugin_lang_get with new gettext approach - David Hicks04:30
Paul_46hi06:24
Paul_46my mantisbt2 branch is slowly coming along nicely06:24
dhx1Paul_46: can you push the branch to GitHub again?06:30
Paul_46what does error: src refspec does nto match any mean?06:36
Paul_46done06:38
dhx1ok :)06:48
dhx1have you been following 'next'?06:49
Paul_46a little06:50
Paul_46I dont like the idea of gettext atm06:50
Paul_46at least not yet06:50
dhx1any particular reasons?06:50
Paul_46I think we need to tidy up other stuff first06:50
Paul_46i.e.06:50
Paul_46try and get some of the stuff moved to objects06:51
Paul_46think about templates06:51
dhx1I'm hesitant about objects and templates06:51
Paul_46rather then change the language layer, then think about templates and decide we want a different layet06:51
Paul_46i'm currently just seeing where I can take this branch06:51
dhx1My attempt at using gettext is to provide a basis for proper localisation/internationalisation support when we get around to fixing the UI06:52
dhx1if I had to propose a "template" system, it'd probably be a DOM object we manipulate within PHP using DOM function calls06:53
dhx1... where we try to avoid doing that wherever possible06:53
Paul_46i'm just thinking more and more that if you look at mantis06:54
Paul_46and some of the bugs that come in06:55
Paul_46and that we fix06:55
Paul_46some of them are due to the way we structure stuff atm06:55
dhx1indeed06:55
dhx1mostly it's due to having multiple ways of doing the same action in MantisBT06:55
Paul_46so I don't mind taking a sledgehammer approach and seeing where I end up06:55
dhx1SOAP API has different access checks than the standard HTML interface06:55
dhx1etc06:55
Paul_46yes06:55
Paul_46[part of the reason i'd kinda like objects is such that it helps us define (Eventually) an interface06:56
Paul_46i'd like to see soap api bascially be:06:56
Paul_46function soap_get_user_name($user_id ) {06:56
Paul_46return MantisUser($user_id)->get_username();06:57
Paul_46where as, atm, it's probably running it's own sql query :P06:57
Paul_46to avoid the fact that the get username call in the html interface formats the username in bold text or something :P06:57
dhx1I partially agree06:57
dhx1but loading an entire MantisUser object from the database just to get the username is wasteful06:58
Paul_46I partially agree :)06:58
Paul_46there's a two fold question there06:59
Paul_46atm, I suspect we do select *06:59
Paul_46and cache06:59
Paul_46you like the idea of doing select name06:59
dhx1so I'm leaning more towards having a user_get_name($userID) function that is used whenever you only want to get a user name from an ID06:59
Paul_46one could make the object lazy load data06:59
dhx1hmmm06:59
dhx1lazy load is too slow/cumbersome/complex07:00
Paul_46however, my view is that we want to minimise db queries07:00
dhx1I'd rather just load the whole object once07:00
dhx1do we really need to minimise database queries?07:00
Paul_46so if you are going to call user_get_name and then later on the page user_get_realname07:00
Paul_46i'd say it's a 'bug' to call that as two queries07:00
Paul_46i'd rather have07:00
dhx1any decent SQL server will cache query results...07:00
Paul_46a) select name, realname07:00
Paul_46or select *07:00
dhx1I agree07:00
Paul_46sure but if the sql server isn't on the same box it's still a tcp roundtrip07:01
dhx1which is why we still need a User object07:01
dhx1for cases where we're working on just one user at a time and need to access many different fields07:01
Paul_46and arguably if there's only 30 bytes of data in the column, it's probably less load on the sql server to get 30 bytes first then to get 4 bytes and 4 bytes later in two questions07:01
dhx1actually -- that's debatable too in some ways07:01
dhx1agreed07:01
dhx1but we could easily do a query builder approach07:02
dhx1create the query with all required fields07:02
dhx1and then execute the query07:02
dhx1and output that data to the user07:02
dhx1it's not quite that easy though :(07:02
Paul_46seperately, i've been working on some other projects for game i play07:03
dhx1I'm thinking of redoing the Enum stuff next07:03
Paul_46that use memcached to cache data07:03
dhx1hmmm, memcached is still useful?07:03
dhx1why can't the SQL server provide that caching for you?07:03
dhx1it seems OK to use for data that doesn't need to meet ACID requirements07:04
Paul_46i think it's a case of simplicity07:04
Paul_46think about it07:04
dhx1I don't see the point myself07:04
Paul_46if you wrote some code that said07:04
Paul_46select name whre id = 1 from mysql07:05
dhx1I'd rather use a front end caching layer such as Varnish (microcaching)07:05
Paul_46the mysql servers going to have to do some work07:05
Paul_46check permissions etc07:05
dhx1cache pages on the front end for 0.5 seconds to limit PHP being called at all to 2 times/second07:05
Paul_46wheresa with memcached your basically going07:05
Paul_46"gimme this id from ram"07:05
Paul_46and getting back whatever you've stored07:06
Paul_46so if that's a json/serialized object07:06
Paul_46that's gonna be faster then going to mysql and building the object07:06
dhx1SQL server cache will double check if the value has changed, permissions are still valid to access the data, etc07:06
Paul_46microcaching is good for readonly data07:06
dhx1so that is all you're avoiding07:07
Paul_46yea, still my point is that's gonna be technically slower then the memcache check07:07
dhx1a valid point in some cases :)07:07
dhx1microcaching is good for pages served to anonymous users (read only)07:07
Paul_46I was trying to work out if I had nginx setup right to do microcaching07:07
Paul_46but then I realised07:07
Paul_46well two things07:07
dhx1which is the bulk of traffic to an open source bug tracker07:07
Paul_46a) I couldn't work out whether i had the micro caching doing it :)07:07
Paul_46b) I realised that the page content changes if your logged in, so probably shouldn't cache it :)07:08
dhx1we'd have to redesign MantisBT to work with microcaching07:08
dhx1CSRF tokens cannot exist on pages that are microcached (or else they quickly lose their security value)07:08
Paul_46tbh, i'd kinda wonder if microcaching would work for a bug tracker07:08
Paul_46i.e.07:08
Paul_46microcaching is normally "lets store this page for a minute or so" right?07:09
dhx1more like <1 second07:09
dhx1the idea is that the user won't notice the effect of cachine07:10
Paul_46yea, so you'd have to have a really busy bug tracker for it ever to kick in07:10
Paul_46:)07:10
dhx1*caching07:10
dhx1but you can pump through 10,000 hits/second :)07:10
dhx1right07:10
dhx1still good for open source bug trackers that have bugs hit the front page of a popular website07:10
dhx1funny bug reports, major security vulnerabilities, "how to do XYZ", etc07:11
Paul_46the one thing i've found with the memcache thing07:11
Paul_46is I think it does make a difference07:11
Paul_46for example, the code i'm working on memcache's sql queries [ personally i'm not sure i like this as i'd rather cache objects etc]07:11
Paul_46however, it'll take 10seconds to generate the page running the sql queries07:12
Paul_46if you turn on the memcache caching, it does speed up07:12
dhx1are you using PDO?07:13
Paul_46this code doesn't07:14
Paul_46looking at http://dev.mysql.com/doc/refman/5.1/en/query-cache.html07:14
Paul_46"Be cautious about sizing the query cache excessively large, which increases the overhead required to maintain the cache, possibly beyond the benefit of enabling it. Sizes in tens of megabytes are usually beneficial. Sizes in the hundreds of megabytes might not be."07:14
Paul_46---07:15
Paul_46Caching full queries only – Meaning it does not work for subselects, inline views, parts of the UNION. This is also common missunderstanding.07:15
Paul_46ahh07:15
Paul_46some of the code i know uses unions/subselects07:15
Paul_46so maybe it's not caching07:15
Paul_46Table level granularity in invalidation – If table gets modification all queries derived from this table are invalidated at once.07:16
Paul_46yea right , ok07:16
Paul_46this code basically is for a killboard07:16
Paul_46so every 30 mintues or so, new data comes in07:16
Paul_46== cache wiped07:16
dhx1Paul_46: queries with UNION, etc will still be cached... in full07:17
Paul_46nod07:17
Paul_46whereas, with memcache, there's say 20,000 'corporations'07:17
Paul_46those objects stay in cache all the time07:17
Paul_46and it just updates the cache object when updating the db07:17
dhx1if you're able to cache stuff for 30 minutes... great!07:17
dhx1I'd be happy to cache stuff for 1 minute hah07:17
dhx1we need to work out whether we need ACID compliance or whether it can be "eventually consistent"07:18
Paul_46btw, also in same webpage: "It is not that fast Query Cache is fast compared to running the queries but it is still not as fast as specially designed systems such as memcached or local shared memory. "07:19
Paul_46but then we know this07:19
Paul_46the reason db's have query caches is to speed up common queries07:19
Paul_46but if you specifically design to store something in ram only, it's well :)07:19
dhx1I can see where memcached would be useful for MantisBT07:21
Paul_46i'd probably store projects, users, permissions, config in memcached07:21
Paul_46and probably let bugs/history hit db07:21
dhx1but I wouldn't use it if SQL server cache or front end microcaching can take the load instead07:21
dhx1not keen on that...07:22
Paul_46that's where you'd probably see a nice performance gain07:22
*** Quits: siebrand (~siebrand@5353A6DC.cm-6-4c.dynamic.ziggo.nl) (Quit: siebrand)07:23
*** Joins: siebrand (~siebrand@5353A6DC.cm-6-4c.dynamic.ziggo.nl)07:23
Paul_46i.e. you'd instantly have the username etc available07:25
Paul_46but i'd still make the point of07:25
Paul_46I wonder how many bug trackers get enough traffic07:25
Paul_46to need that or microcaching07:26
Paul_46:)07:26
dhx1right :)07:26
dhx1I don't like fixing problems that don't exist07:26
Paul_46at the same time, if you got a user object, it would be trivial to allow that to be saved to ram/disk if required at some point07:27
dhx1it'd be better to try and optimise our queries (and database design) before worrying about caching07:27
Paul_46well I merged the bug_text tables together07:27
Paul_46to get rid of the join07:27
Paul_46again, that's partly on the basis it's easier to map to objects at that point07:28
Paul_46i'm just really at the point of trying to take a fresh look at things07:28
dhx1agreed on that merge07:28
dhx1same07:29
Paul_46i'm not really fussed if my branch<>next<>1.2 get diverged07:29
Paul_46as thats just a case of going through07:29
dhx1I'll probably cherry pick from your branch :)07:29
Paul_46i've been cherry picking form yours/1.207:29
Paul_46I think the main thing is, within a two months, we probably need to sit down and review what branches look like07:30
dhx1yep07:31
Paul_46the only thing I dont like is there's been some commits to do things like "add a new 'g_allow_permanent_cookie'  config variable"07:32
Paul_46in that particular one i'm thinking two things07:32
Paul_46a) I'm getting to the point I dislike that we have 200000000 config's07:33
dhx1I've been including everything from 'master' so 'next' can be merged back in easily07:33
dhx1I could merge it right now in 5 seconds if I wanted to07:33
dhx1it may be the case that some of this new functionality is removed in 'next' :)07:33
dhx1a - agreed07:33
Paul_46b) the additional of that config variable kinda makes me want to function the plugin framework idea, deprecate that value and go "you want that feature, make a authentication plugin"07:34
Paul_46[and maybe even provide said plugin]07:34
dhx1I feel like I'm being forced by MantisBT into redesigning custom fields, enumeration types, etc07:34
dhx1the more I think about it the more I prefer these definitions occur within .php files (not in the database)07:34
Paul_46I reverted some commit locally07:35
dhx1party because of gettext benefits of being able to parse raw .php files07:35
Paul_46that added a extra text field to some custom field table07:35
Paul_46you know custom fields have 'display on report', 'display on open' etc?07:36
dhx1yes07:36
dhx1I'd rather they didn't...07:36
Paul_46and i'm pretty sure we had a long discussion at time about whether we should store that information in table07:36
Paul_46or as seriliazed data etc07:36
Paul_46given way things are going now a days07:36
dhx1I'd rather we didn't store custom field definitions in the database07:37
Paul_46i'm kinda thinking, i'd rather have that stored as some json encoded string [yes, json is slower for one of decode/encode then serialize]07:37
Paul_46oh, I think i see what your saying07:37
dhx1I'd rather we have something like:07:38
Paul_46are you saying if someone wants a custom field exactly 20 characters long, you have a. php file defining that?07:38
dhx1fields/Reporter.php07:38
dhx1fields/Resolution.php07:38
dhx1etc07:38
dhx1and each file provides all the definitions/etc needed to make the field work07:38
dhx1right07:38
dhx1OO/traits could be used to inherit from other classes07:39
dhx1so it'd be easy for users to take an existing definition and slightly modify it07:39
Paul_46well, i'm kinda 'against' that or well07:39
Paul_46fields/file.xml/json ==> 'ok'07:39
Paul_46i'm not sure aobut .php as that kinda implies end users need to code to do stuff07:40
dhx1the main reason I'm looking at this approach is for internationalisation reasons (it's better to store custom field names, etc in .php files so gettext can get to them)07:40
dhx1gettext methods for translating custom field details allows us to provide translation hints, pluralisation support, etc07:40
Paul_46thing is, that's not user friendly :)07:40
Paul_46for example, at work, i've allowed my boss to add some custom fields07:41
Paul_46he doesn't do coding07:41
dhx1we can provide a GUI frontend to make it easy for users to create new fields...07:41
Paul_46then you've got gui writing .php files, which could in theory be a risk07:41
Paul_46however, tbh, even if you had07:41
Paul_46fields/Reporter.json for example, you could still translate it later on07:42
Paul_46more importantly07:42
Paul_46have you ever looked the 'thebuggenie'?07:42
dhx1looking now07:42
Paul_46http://issues.thebuggenie.com/07:42
dhx1looks nice, thanks for the link07:45
dhx1I don't agree with some of their UI decisions07:45
Paul_46nod07:45
dhx1for instance, tabbed Comments, Attachments, etc07:46
Paul_46at the same time, i'd probably rather look at that then some bits of mantis07:46
dhx1rather than just a long linear page07:46
Paul_46)07:46
GitHub18[mantisbt] siebrand pushed 1 new commit to master-1.2.x: http://git.io/OA1B6A07:46
GitHub18[mantisbt/master-1.2.x] Localisation updates from http://translatewiki.net. - Siebrand Mazeland07:46
Paul_46actually07:46
Paul_46if you think about it07:46
dhx1siebrand: thanks :)07:47
Paul_46the tab stuff is probably more user friendly07:47
dhx1Paul_46: it's hiding content behind mouse clicks :(07:47
Paul_46i'd probably put the attachments/comments on first page07:47
Paul_46then hide07:47
Paul_46or well07:47
Paul_46having our bughistory hidden07:47
dhx1in UI terms, "discoverability" isn't very high07:48
Paul_46would be a good thing07:48
Paul_46i'm not sure about hiding related issues etc07:48
dhx1these are things which user CSS stylesheets can override (replacing a lot of our bad settings!)07:48
Paul_46but as a whole07:48
Paul_46http://issues.thebuggenie.com/thebuggenie/issues/new/bugreport07:49
Paul_46I like that07:49
dhx1issue history is very important to show07:49
Paul_46wysiwtg wiki editor07:49
dhx1I'm not sure whether we want to keep it in a separate table07:49
dhx1or mix it in with the comments (my preference)07:49
dhx1there are some other ideas I'd like to look at including:07:50
Paul_46i'd like to come up with a mobile version of mantis in core07:50
dhx1* bugs don't have owners (reporters) -- a bug will often affect more than one user and they can work together collaboratively to describe and fix the issue07:51
dhx1* formal state machine implementation of workflows07:52
dhx1* integration with mailing lists (have you used debbugs?)07:52
dhx1* pick-n-mix workflows/field types/etc that come bundled with MantisBT by default... administrations can choose different development models in a single click07:53
dhx1* full text search07:54
dhx1* more emphasis on graphical representations (relationship graphs, critical path analysis, statistic plots, etc)07:55
dhx1... all aspirational 2030 goals of course :)07:55
Paul_46nod07:56
Paul_46:)07:56
dhx1on a different matter, have you used Firefox's new 3D HTML display?07:58
Paul_46no?07:58
Paul_46whats this07:58
dhx1Tools -> Web Developer -> Inspect -> 3D (M)07:59
dhx1it shows the layers of a web page (nesting) in 3D07:59
dhx1mantisbt.org/bugs has 2 logos on top of each other in different layers08:00
dhx1perhaps that's just how this 3D view renders it sorry08:01
Paul_46kinda intersting though08:02
Paul_46anyway, main thing I want atm is a free opensource html5 layer for mantis08:03
Paul_46:)08:03
dhx1new UI?08:04
Paul_46I want itouch version :P08:04
Paul_46so I can use it at work easily08:04
dhx1ah :)08:05
Paul_46hoping to find some work time for mantis soon08:05
Paul_46we just finished windows 7 deployment prettyy much08:05
Paul_46boss is away for a week in may/june08:05
Paul_46forget which08:05
Paul_46i have ~50-60 open issues in mantis at work08:06
Paul_46which is down from >10008:06
Paul_46and a whole bunch of those 50 are just sitting there08:06
dhx1:)08:06
Paul_46for example, one was 'lets try and install atmosphir' - which is some game thing that's got mentioned as maybe being educational08:07
Paul_46but from what I can tell, it's dieing a death08:07
Paul_46i.e. they went offline for a week or two a month ago08:07
Paul_46and restored website/forum back 6 months08:07
Paul_46the app downloads but won't run propelry for me08:07
Paul_46the guy running it says new dev team coming on in may/june08:08
Paul_46and yet, some of the "old staff" on forums are like posting "yea, sod this, we're off" :)08:08
Paul_46so that's 1 of 50 that isn't going anywhere08:09
Paul_46:)08:09
dhx1hah08:09
dhx1argggh08:25
dhx1history_localize_item with plugin fields is an ugly problem08:25
dhx1after a plugin is uninstalled, localisation for plugin history entries becomes unavailable08:26
Paul_46I still think we need to take a step back :)08:26
dhx1this is actually quite an ugly problem... how do we store history of a bug such that it is usable even after the plugin is uninstalled08:28
dhx1I do have a solution but it's a bit of work08:28
Paul_46well, the question would be if a lpugin gets removed, should the histroy get removed08:29
dhx1Solution #1: If the plugin is removed then all the data associated with it (including history) is removed08:29
dhx1Solution #2: When a plugin is installed it updates a user-configurable text domain that is permanently moved around with the installation08:31
dhx1both approaches are bad... but I can't see an alternative ;/08:32
dhx1solution (2) essentially leaves part of the plugin behind "forever"08:32
dhx1I prefer solution (1) on the basis that it promotes a clear separation of duties between MantisBT core and plugins08:34
Paul_46disabling/enabling plugin and uninstalling are two different things remember :)08:34
dhx1yeah08:35
Paul_46in fact, thinking about it, what your partly describing may be something i'd thought about in terms of logging/notification08:35
dhx1true08:35
dhx1same problem08:35
Paul_46i.e. atm, we trigger an event and let plugin do whatever08:35
dhx1because 'history' is really just a log08:35
Paul_46i'd like to [for notifications], let plugin register a notification hook08:35
Paul_46at that point, uninstalling would let us see in core 'did this plugin have any hooks registered'08:36
dhx1yep, item #59 on my TODO list is to add a new notification subsystem to MantisBT :)08:36
Paul_46if yes, there should be a cleanup function08:36
Paul_4659?08:36
Paul_46random number? :P08:36
dhx1(or some other high number :P)08:36
Paul_46hehe08:36
dhx1I wonder if anyone has provided localised logging before... hmm08:37
dhx1we could store a log for each locale :P08:37
Paul_46how about 'no'08:37
dhx1I'm actually keen to rethink "history"08:39
Paul_46this is partly where i like my 'objects' approach08:40
dhx1perhaps we should have a "bug_revisions", "project_revisions", "user_revisions", etc table for each MantisBT table08:40
Paul_46i'm assuming it'll need to be rewritten08:40
Paul_46or well, re-organised08:40
Paul_46but it gives option to go through code base and move code around :P08:40
Paul_46if you noticed, I'd move all of the manage_* stuff in my branch to a /manage bit08:41
Paul_46on the basis that if we do a new UI for end users08:41
Paul_46we should probably work on the front end side first08:41
Paul_46and if admin bit stayed with old api08:41
Paul_46that would be fine08:41
Paul_46but by seperating it out, it's easier to track how many pages we've got and what they do ;p08:42
dhx1yeah08:43
*** Quits: dhx1 (~anonymous@60-242-247-232.static.tpgi.com.au) (Quit: Leaving)17:48
*** Joins: giallu (~giallu@fedora/giallu)18:16
*** Quits: Paul_46 (~IceChat09@cpc1-enfi15-2-0-cust580.hari.cable.virginmedia.com) (Quit: I used to think I was indecisive, but now I'm not too sure.)19:05
*** Quits: giallu (~giallu@fedora/giallu) (Ping timeout: 276 seconds)19:44
*** Quits: sdfjkljkdfsljkl (~sdfjkljkd@static.96.23.63.178.clients.your-server.de) (Remote host closed the connection)20:00
*** Joins: sdfjkljkdfsljkl (~sdfjkljkd@static.96.23.63.178.clients.your-server.de)20:00

Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!