Helena Edelson

Akka, Spark, Scala, Cassandra, Big Data, Data Science, Cloud Computing, Machine Learning, Distributed Architecture

  • cloud

  • RSS Subscribe

  • Topics

  • Archives



    • @helenaedelson on Twitter
    • Helena Edelson on LinkedIn
    • GitHub Octocat

      helena @ github

      • Status updating...


Archive for the 'AOP' Category

An Agile Software Development Lifecycle – Increasing Efficiency and Throughput, Decreasing Bottlenecks

Posted by Helena Edelson on 25th February 2012

 

This post is in progress, and comes from years of finding things that work, as well as slamming into the pain and not having the authority to change the process (That’s called using power for good, people), or saying, “Good luck with that”. One thing I have learned in all the org’s I have frequented is that most orgs that have problems recognize this and sincerely want and are trying to improve the process. Getting the process right is an art.


SDLC – Increasing Efficiency and Throughput, Decreasing Bottlenecks


Whether you are new to Agile or seasoned, you should read The Agile Manifesto – poetry for the weary.

 

Software development lifecycle improvements can increase throughput, manageability and maintainability, vastly increase overall efficiency, and decreases bugs. A major factor is improving the agile process for the organization (from planning to deploys), for which you need to not only educate staff but also have employee buy-in to the process. Jumping into this with a system that is in need of re-architecting could create more bottlenecks and technical debt so timing and rollout of new processes must be planned well.

 

Planning And Requirements


Insure that requirements are broken down into what can be completed and demoed correctly in each sprint. Scoping each unit of work for a feature, enhancement, bug fix or task must be done in such a way that dependencies are never a bottleneck and goals are achievable and testable. The business should set requirements in a scope that is achievable, partitioning work sometimes over multiple sprints.

 

Tooling


Do not overlook a very important part of the process that lends to success – your process tooling. Tools like JIRA are often underutilized. Leveraged properly, it makes the work and process transparent, improving not only efficiency but accuracy of implementations, testing, and deploys. If you don’t already have Greenhopper for JIRA, get it. JIRA is not just a bug tool, it is a SDLC tool.

 

  • I would not do a project without JIRA (with Greenhopper)
  • I would not do a project without GIT as the repo. Anything else and you are asking for pain and incredible inefficiency

Turning Requirements Into Stories


This may be the starting directive: Build in real-time asynchronous stock ticker updates to the client. In Agile, the Story should look like this:

 

As a User I want to receive real-time updates of stocks I have selected to watch, so that I can take immediate, accurate advantage of market fluctuations

We can not proceed without covering dependencies. A functional dependency is something that requires or depends on another thing being done first. Example: I want to tweet, I have a dependency on an internet connection to send the tweet in order to accomplish that task.

This assumes or highlights many things, for instance, what could be done in parallel as tasks of the same story? What may more appropriately completed in a prior Sprint?

  • Is there a UI dependency to create a form where the user can select stocks to watch, submit the form?
  • Does a new db table need to be created for persisting stock selections by user?
  • What in the messaging system needs to be added to do this?
  • What could be done in parallel as tasks of the same story?
  • How can this be architected to allow for consecutive sprint development?
  • How can QA best test this over each sprint until the final task is committed?
  • Do we put this in a feature branch or roll each part in by sprint so QA can test?

A business dependency, which may also be called a requirement, is something engineering needs from the business in order to complete a Story or Task. For example:

As a business owner I want to authorize all incoming user requests from the client to insure that user has been granted rights to do what they are requesting to do in the system.

Say the Product Manager brings this up in a Sprint Planning meeting to the team, reviews the requirements, and one of the engineers says, “I see a requirement missing, there is no failure scenario”. If the client is simply making API REST calls to the server, then any client can call what it is not granted authority to do. A failure plan is needed for what is generated on the server and sent to the caller/client and if the client is a UI, how this is displayed. This could need a new UI task and time.

 

Story Points And Time Boxing


Possibly the most highly-debated part of any SDLC process methodology, whether agile, lean or other.
Planning Poker is an interesting read.

Planning And Effective Use Of Tooling


First, a Project Manager should identify the availability in hours per day of each team member for a sprint to get the total hours of the team for a sprint. I should note here that I have fought with QA many times over the years and may be biased.

 

  • Product Manager Creates a Story ticket that will have child tasks
    • The business insures that business requirements are accurately and granularily described in the story’s description
    • Team (Dev/QA) assesses the business requirements, and identifies questions or information lacking by the business
    • Project Manager assigns the Story ticket to the business/Product Manager to complete – and everyone can see where in the process this ticket is. When complete, the business/Product Manager assigns ticket back to Project Manager
    • Product Manager and Project Manger reviews with team
      • If team deems requirements are adequate to proceed, Project Manager can schedule the Story
    • Team (Engineer(s) – Dev) votes on the number of story points to assign to the Story – how many days will it take to complete? What is the difficulty of the story?
    • Does this story have dependencies?
      • If so, are they completed or scheduled? How long will they take?
      • Does this story require an architect before developers come on?
      • Do we need to prototype or vet a new technology decision or strategy first?
      • Is there a functional or UI dependency?
      • Dependencies in the same sprint must be scheduled in such a way that no
    • Team (Engineer(s) – Dev) create Child Tasks under Story – if a junior team, Team Lead insures accuracy and that nothing is left out
      • Developers task out what needs to be done to complete a scoped story
      • Developers pick task tickets and assign to self
      • Developer / QA timeboxes task – how many hours will this take?
    • Team (Engineer(s) – QA) create Child Tasks under Story
      • Create task ticket to create test plan for story
      • Review test plan with the business and developers
      • Attach approved test plan to Story’s JIRA ticket
    • more coming

Now insure that the hours it will take to complete the work is achievable given the availability of the team.

 

Development – and Effective User Of GIT


  • TDD – Test Driven Development
    • 1:1 correlation between requirements from the business and unit/integration test methods
    • Unit test: testing a method or small unit of code
    • Integration test: testing a Service method that covers multiple component (DAO, Util, Messaging, etc)
    • Tests must be written in such a way that they are OS and Environment independent
    • Tests must be written in such a way that they can be run by an automated process – Continuous Integration (CI)
    • If CI tests fail, the team should be notified by email that tests have failed, and attended to quicky
    • Before code check in
      • Latest changes in the current branch should be pulled in
      • The new code must compile with the latest code in the branch
      • Tests should be run by the developer and pass
        • This means that every developer is responsible for all of their tests passing when a team mate checks out the latest code
      • New code must be deployed successfully by the developer
  • Coding Standards
    • Every org needs coding standards and a means to insure that such standards are being implemented
      • Code Reviews
      • Developer training
      • Best Practices instruction
      • Lunch and Learn presentations for the team
      • Team education of the system, tools, processes – the more everyone knows, the faster and less error prone things will be
      • How to properly test code
    • Architecture enforcement with AOP in codebase
      • Automate catching poor architecture or code implementations in dev-time
        • No engineer has time to stay on top of static code standard documentation

Technical Debt

If an org is accumulating technical debt in part due to the business: sales and marketing pressures for new features and bus fixes faster than engineering can also take care of issues that need to be resolved as well as doing work properly the first time, then the risk management of the system is put on the back burner and accumulates. This will eventually blow up.

Technical debt time must be built into R&D sprints, and often are done as sprints on their own.

 

QA, Staging, Deploys, and Rollback Strategies


Not a free-for-all, I have not written this yet. However, deploys should be automated. If you need to do it manually, there is something wrong with your application architecture and/or deployment architecture.

Share/Save

Posted in Agile, AOP, Software Development, Software Development Lifecycle | No Comments »

Transaction Isolation and Spring Transaction Management

Posted by Helena Edelson on 21st October 2009

I was asked a question by a Spring student of mine and as it pertains to Spring Transaction Management, as well as transaction  management and databases in general I thought I’d share it with everyone if it may help anyone else. The question went something like this:

How do we prevent concurrent modification of the same data? Two users are in the middle of a transaction on the same data. How do we isolate those operations so that other transactions can not read the data and how to we handle synchronizing  changes to that data with commits? This deals with preserving  data integrity, the underlying locking mechanisms of your database, transaction demarcation and transaction isolation settings which are configurable by Spring Transaction Management but ultimately controlled by your specific database vendor’s implementation.

Some ways to think about transaction isolation are:

  1. The effects of one transaction are not visible to another until the transaction completes
  2. The degree of isolation one transaction has from the work of other transactions

So how do we keep one transaction from seeing uncommitted writes from other transactions and synchronize those writes?

Remember ACID = Atomicity, Consistency, Isolation and Durability?

The SQL standard defines four levels of transaction isolation in terms of three phenomena that must be prevented between concurrent transactions. These phenomena are:

Dirty read: A transaction reads data written by a concurrent uncommitted transaction.
Nonrepeatable read: A transaction re-reads data it has previously read and finds that data has been modified by another transaction (that committed since the initial read).
Phantom read: A transaction re-executes a query returning a set of rows that satisfy a search condition and finds that the set of rows satisfying the condition has changed due to another recently-committed transaction.

For most databases, the default transaction isolation level is “read committed” (your transaction operates on the data that it sees at the beginning of the transaction). You can, for example, configure transactions to be serializable which increases lock contention, but is more suited for critical operations. Here we could get into database locking, predicate locking, etc. While it is outside the scope of this post, it is very worthwhile to know; I do suggest reading up on it.

In the java layer, here are the standard isolation levels defined in the JDBC specification, in order of weakest to strongest isolation, with the respective inverse correlation on performance:

• TRANSACTION_NONE: transactions are not supported.
• TRANSACTION_READ_UNCOMMITTED: dirty reads, non-repeatable reads and phantom reads can occur.
• TRANSACTION_READ_COMMITTED: dirty reads are prevented; non-repeatable reads and phantom reads can occur.
• TRANSACTION_REPEATABLE_READ: reads and non-repeatable reads are prevented; phantom reads can occur.
• TRANSACTION_SERIALIZABLE: dirty reads, non-repeatable reads and phantom reads are prevented.

Spring’s Transaction Management Isolation levels:

ISOLATION_DEFAULT
Use the default isolation level of the underlying datastore.

ISOLATION_READ_COMMITTED
Indicates that dirty reads are prevented; non-repeatable reads and phantom reads can occur. This level only prohibits a transaction from reading a row with uncommitted changes in it.

ISOLATION_REPEATABLE_READ
Indicates that dirty reads and non-repeatable reads are prevented; phantom reads can occur. This level prohibits a transaction from reading a row with uncommitted changes in it, and it also prohibits the situation where one transaction reads a row, a second transaction alters the row, and the first transaction rereads the row, getting different values the second time (a “non-repeatable read”).

ISOLATION_SERIALIZABLE
Indicates that dirty reads, non-repeatable reads and phantom reads are prevented. This level includes the prohibitions in ISOLATION_REPEATABLE_READ and further prohibits the situation where one transaction reads all rows that satisfy a WHERE condition, a second transaction inserts a row that satisfies that WHERE condition, and the first transaction rereads for the same condition, retrieving the additional “phantom” row in the second read.

Note the Spring isolation and JDBC isolation levels are the same.

So for XML configuration of transaction demarcation in Spring:

<tx:advice id=”txAdvice” transaction-manager=”transactionManager”>
<tx:attributes>
<tx:method name=”insert*” read-only=”false” propagation=”REQUIRED” isolation=”READ_COMMITTED”/>
</tx:attributes>
</tx:advice>

And for Annotation configuration:

in xml: <tx:annotation-driven/> to enable @Transactional

in your service layer,  on the class or method level:

@Transactional
public class DefaultFooService implements FooService { ...} 
@Transactional(isolation = Isolation.SERIALIZABLE)
pubic void createSomething(..){ ... }

Share/Save

Posted in AOP, Java, Spring, Transaction Management | No Comments »

JPA 2.0 Concurrency and locking and AOP Around Advice for Deadlock Retry

Posted by Helena Edelson on 20th July 2009

This article by Carol McDonald from Sun on JPA 2.0 Concurrency and locking is interesting. I like the idea of using AOP for handling crosscutting concerns such as Deadlocks, such as this:

1. You define your Deadlock Aspect
2. Write your @Around advice for stateless or idempotent operations that throw HibernateException from classes marked @Repository:

Around any idempotent operation, such as in a stateless business service, when a DataAccessException is thrown, keep track of how many tries, and re-execute:

@Around("idempotentOperation()")
public Object retryDeadlockLosers(ProceedingJoinPoint pjp) throws Throwable {
int attempts = 0;
DeadlockLoserDataAccessException looserEx = null;
while (attempts++ < maxAttempts) {
try {
return pjp.proceed();
} catch(DeadlockLoserDataAccessException ex) {
loserEx = ex;
}
throw loserEx;

}

Which is more a pattern you can use wherever needed and how such as you can apply to go to another rack, hop to another database, what have you.

Share/Save

Posted in Annotations, AOP, Java | No Comments »

SuMQ Messaging Framework

Posted by Helena Edelson on 24th March 2009

I’ve just started staging my new project, SuMQ, and appreciate patience as this will take me a while to get the code standardized and on google.

SuMQ is a light-weight enterprise messaging framework built in Java, leveraging Spring, JMS, and ActiveMQ. It plugs into Flex Messaging via BlazeDS for the client. This can also be configured for other clients aside from Flex.

The sample will be ready for clustered BlazeDS instances and load balanced Application Servers.

code.google.com/p/sumq

Share/Save

Posted in ActiveMQ, Annotations, AOP, Application Cofiguration, BlazeDS & LCDS, Broker Topology, Java, JMS, JMS Broker, Messaging, Spring, Spring JMS | No Comments »

 
a2a_init("page"); a2a_init("page"); a2a_init("page");