Author Archives: Etienne Bley

Think customer focused when designing – work yourself out a job

Posted on by

If you think bottom line for your customer it will be noticed. At Telvent, I worked not only on projects but also how to make design itself better and faster so that Telvent could hire lower paid designers to use the higher paid engine design that I developed. This creates a demand for what you are doing since you are so far ahead of the curve because your designs start designing themselves that you will always be in demand. It is a win-win. Here are some other tips that keep you looking like a professional and never like a bafoon. One caution however is to market what you are doing. If you don’t the extra time you spend upfront getting the customer better and faster might make you look like you are wasting time. Show the customer how it is a win-win or a triple win which it usually is – the last win is the customer being able to design the next iteration with cheaper staff or in half the time.

At first this seems to take more time, but as you get successive wins under your belt (and hence the customer’s too) it starts to get noticed that you can do just about anything.

Here are some tips for designing in a win-win way

  • think modular design – how can what I do be a class or a reusable part. How can once piece of lego have just one more hook that I can use in the future. Watch out that the complexity stays easy to configure though
    • think toolbelt. The more tools you have the faster and more stable was the design
  • document document document your code. Blog so-to-speak what you do day to day and explain to yourself what you are trying to accomplish. In the immediate it helps you even explaining it to your customer or others who interface with what you are trying to accomplish, in the long run it helps your customer
    • for .net for example, learn the sandcastle documentation rules. It takes an afternoon and about 4 days of using it and you have all the rules in your head. Keep it up or you will start to lose it
    • learn the rules for php doc, Java doc or what ever else you are using to document your code so that a machine can cobble it together
    • document on the side with Word (words and final pics) and Powerpoint (generating pics) since they are so prolific and seemingly universal. You can use proprietary software but Work and Powerpoint are everywhere your customers are and hyperlink quite well
    • use wiki’s for communication between collegues – they can edit the doc to be even better than you alone can accomplish – the power of  collaboration
  • Use a code repository. I have used SVN, GitHub and Mercurial. Using a back up program does not cut it – I have tow of my own personal prize apps for iOS – luckly these were personal projects
    • SVN is the best for versioning binary files since you can lock others out while
    • GitHub and Mercurial are great for multisite collaboration or a regimented distribution system to the end customer. Slow and steady wins the race.
    • be verbose in your comments
    • never procrastinate with getting code into a repository and schedule time at then end of each day. Once a VM went on the fritz and many fellow coders had to redo the code – not so with my code – thank goodness I followed the golden rule
  • Never hard code – no matter what the pressure. If you do – use the well known phrase “TODO” in your code and schedule a time in the future to work this code from hard coding to dynamic design. This makes you look like your intentions were pure which they are. Document when you hard code and mention ideas to work this out to a config file or config-array in the future
  • keep todo lists
  • keep people informed of what works and what is left to go in black and white. This is where wiki’s or published excel sheets are so handy to clarify what is done and what is to go
    • write down – I repeat – write down where struggles are occurring so others can help with experience or with pulling strings to get items out of your way
    • never ever assume the other person has heard you – it takes 7 times on average before someone ‘gets what you are really saying’
    • keep track of your hours and publish/communicate them frequently
  • put larger goals aside to get quick wins for you and your customer/boss. If it is important to them, it should be important to you. Agree to disagree sometimes and get items done that satisfy others if they deem it important

These are some of the basics of programming and half has nothing to do with coding itself but the practice of communication, politics and being organized. If you practice these thing then good things will come.

Please add comments and I will work them into the text above for others.

Web Scraping – how to do it and add stability

Posted on by

If there is no API available to get info off of a web system – what do you do? You buy a scraping tool or … when that does not work because it can’t get past security or because it is a living website – you MAKE YOUR OWN. A young programmer asked me how to do this. Here is some advise on how-to and what not-to (do). This system that gave me most of my experience successfully scrapes gigabytes of info each night between initiating from a windows machine and uses another host on linux to do the scraping. The system babysits itself with 2 points of reference so when one machine is down – you are informed of it via email.

Idea 1- cURL (discussion follows)
Idea 2- learn how to automate internet explorer (for Windows) or Applescript (for Mac)

cURL is your best friend. It can access web pages with all sorts of security – it is free to use and integrated everywhere it seems. However don’t irritate facbook or kijiji or they will shut your account down. Now I am not myself on facebook because they think I am not really who I say I am – so I made another account and a fake name – I can’t even have a relationship on facebook with my spouse – do they have a ‘divorce app’ on facebook so I can re-marry her from one account to another? They are ok with that. DO NOT SCRAPE CERTAIN SITES – read the fine print.

Back to cURL … it is the root of the engine. Here are some tips when you go to make one

  • there are a million options and they all interact – read, re-read and re-re-read the instruction pages
  • there are lots of programming engines that make api’s. I have used cURL in php, unix command line, perl and windows environments.
  • Google is your friend
  • don’t give up – it can be done
  • save your files to local web files
    • protect these directories if sensitive info is there.
    • delete temporary files

Careful – expect things to change on the website you are changing so …

  • build in cost and expectation for your customers to pay a maintenance agreement. This money should not be considered profit – this is what you will be doing next year when the website is upgraded.
  • use generic names
  • program in blocks – DOCUMENT YOUR CODE for yourself especially (more profit!)

Processing scraped web pages.

  • HTML5 – AHHHHH! HTML5 is not XML – so if you use an XML parser and someone changes it to HTML5 – but … not really since it has lots of XHTML 4 in it . . . did I mention to give your customers the expectation of a yearly fee for this tool?
    • this is late 2012-2013 when I had to wrestle with this – programming languages have not adopted standard HTML5 libraries yet. There is only one pre-alpha library for parsing HTML5.
  • What happens if it is read and there are errors right at the borders of the info you just read?
    • build in retry loops with a slightly bigger (randomly bigger) size and try again
  • how about if the pages are not proper XML and weird things break the parser?
    • simply use a search and replace tool – keep it general with arrays of things to search and replace before it gets parsed
    • finding why XML breaks is a major pain. That is why I like XML because it is exact – play by the rules and all will work.
      • XML Validators are your friend – but don’t trust one – it might have less stringent ones and fool you in thinking all is well – when another will give you a HINT (not the answer) to the area
      • in your code – trap the error and spit out the data that is ‘offensive’ – look before and after it.
      • Do not try to debug it  using the whole file to find incorrect or invalid XML – you might need to recreate an XML header and paste most of the offensive bit out.

That is the only advise I have for now. It was a long while ago when I wrote it – it was supposed to be 30 hours – it was more like 100+ . Price carefully with lots of margin.

Filemaker – a dream tool for dB app prototypes … or real solutions

Posted on by

When I first started using Filemaker – I had to since a customer was enamoured with it. Once I got going I too was enamered. I tried to get another programmer to try it an he too turned his nose up at it, but after using it to prototype – fell in love instantly. Oh- and they have apps for iPad and iPhone – just design new screens with a couple scripts to redirect or use existing screens and just … go!

It is so easy to set up and configure and its programming is done by clicking on commands that to make mistakes is often just not going to happen thereby reducing your design time. The graphical nature of the layouts and picklists and all the other GUI items makes this a dream tool to just get an app/database done quickly. You can share it to clients with Filemaker or iPads witth Filemaker Go or just turn on a (albeit clunky) web rendreable version of your forms/layouts/reports in seconds.

It isn’t cheap. To get Filemaker Server- you should get the Server Advanced with a buy out over $2500 or a a yearly fee of over $200 / a. You cannot write the overhead of the search, add/delete etc. on a web tool as fast as Filemakers’ tools are already ‘just there’. You can customize menus, add graphics tips, tab orders etc.etc in no time. Its what you can do that gets you in trouble as “oh I can add that” for free is such a natural thing to do when making cool tools for people.

Interfacing it . . . takes some trial and error but it can read XML and write XML so with some XSL knowledge it can happen. Fighting with XSL is not always a desired skill to have as hours can go by before you get what you require unless you read your XML book 5 times from font to back. With Filemaker Server Advanced, other interfacing options open up like easy-to-configure ODBC and easy to set up php scripting. If you customer says yes to Filemaker Server Advanced you are good to go.

Buy a copy of Filemaker Advanced if you are going to play with it. It costs about $430 or a yearly licence is $130 on the yearly AVLA. Do not buy FIlemaker Pro to play with the development tools -as you will be missing the step by step debugger and design export tools.

Load webpage, print, load, print, etc. with Applescipt

Posted on by

First, Mac are super cool because beneath that suave exterior is unix. A computer that is easy to use AND super powerful.

But not everything needs unix. Here is the coolest applescript to

  1. load a random bingo card generator which I built in php (the unix part)
  2. print the page
  3. go to step 1 fifty times

[applescript]
set i to 1
repeat 20 times
tell application "Safari"
activate
tell application "System Events"
tell process "Safari"
keystroke "r" using command down
delay 1

keystroke return
end tell
end tell
–open location "http://www.apple.com"
end tell
delay 3
tell application "System Events"
tell process "Safari"
keystroke "p" using command down
delay 0.5
keystroke return
end tell
end tell
delay 4
say i
set i to i + 1
end repeat

[/applescript]