[Open.ogc] Deleted layers still exist?

John Kennedy - NOAA Affiliate john.f.kennedy at noaa.gov
Wed Sep 25 19:58:31 UTC 2013 

Oops.

Communities_2000_NE_SE_Gulf_v4
SI_SE_Gulf_sites_pg_ALLsites
SI_SE_Gulf_sites_pt
communities_pgs
communities_pgs20130506
communities_pgs20130916
communities_pgs20130920
communities_pgs20130924
communities_pts
communities_pts20130506
communities_pts20130916
si_se_gulf_pg_v2
si_se_gulf_pts_v2



John F. Kennedy, PhD, CPG
 GIS Specialist
Contractor: Caelum Corp.
NOAA Fisheries Office of Science and Technology
1315 East-West Highway
SSMC3 Rm 9518
Silver Spring, MD 20910
301-427-8149

Comments? Have I been of assistance? Please let my supervisors know. :~)
Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>


On Wed, Sep 25, 2013 at 2:38 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:

>  Hi John,
>
> I was just about to email you back on this, because I got an update from
> support that unfortunately that feature isn't implemented in GeoServer
> after all.  Apparently he was mistaken because it is implemented in
> GeoTools (the java library that GeoServer uses behind the scenes), but not
> leveraged in the GeoServer code.
>
> I asked what it might take to get that added, so stay tuned.  I'm
> expecting an answer of either 1) you can do it yourself if you want (a
> benefit of using an open source product), 2) it might get added at some
> point, file a feature enhancement request:
> http://jira.codehaus.org/browse/GEOS, or 3) if you have some money, you
> can fund it to be developed by us.
>
> We did #3 when this project got off the ground originally, we funded them
> to develop the workspace access restrictions that allows multiple users of
> the same system, but money has been harder to come by lately (if NMFS has
> some extra lying around, let me know!).
>
> In the meantime, if it's essential to delete the old schemas for those
> tables, I can put a request in to the WOC to see if they can do it manually
> in the database during a maintenance period.  Let me know.
>
> Micah
>
>
>
> On 9/25/2013 2:15 PM, John Kennedy - NOAA Affiliate wrote:
>
> Okay, I deleted 'communities_pgs' using the Web UI and then used hpcc_data_requests.py
> script to load the data. Same result as before. The old schema remains and
> none of the new schema is added.
>
>
>  John F. Kennedy, PhD, CPG
>  GIS Specialist
>  Contractor: Caelum Corp.
> NOAA Fisheries Office of Science and Technology
> 1315 East-West Highway
> SSMC3 Rm 9518
> Silver Spring, MD 20910
> 301-427-8149
>
>  Comments? Have I been of assistance? Please let my supervisors know. :~)
> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>
>
> On Wed, Sep 25, 2013 at 11:39 AM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>
>>  Hi John,
>>
>> I am working on troubleshooting this issue with OpenGeo/Boundless
>> support.  I've gotten the response back that GeoServer should be able to
>> modify an underlying table schema when a new shapefile is uploaded with
>> changes in the attribution, but I think based on your evidence more testing
>> of this is needed.
>>
>> I asked whether it made a difference if the layer (and associated
>> featuretype) were currently published when the new data upload happened, or
>> if they should first be deleted.  If you get a chance, could you test with
>> the same layer you used below 'communities_pgs' by first deleting the layer
>> via Web UI and then doing the data upload (incl new schema) with the
>> hpcc_data_requests.py script?
>>
>> Micah
>>
>>
>> On 9/24/2013 4:14 PM, John Kennedy - NOAA Affiliate wrote:
>>
>> Interesting question. I did a preview and the features do appear to be
>> overwritten in the layer. But the "new" attributes are not loading as the
>> column names defer between the old and the new shape file. I checked a
>> number of polygons and got blank values as shown below.
>>
>>  [image: Inline image 1]
>>
>>  John F. Kennedy, PhD, CPG
>>  GIS Specialist
>>  Contractor: Caelum Corp.
>> NOAA Fisheries Office of Science and Technology
>> 1315 East-West Highway
>> SSMC3 Rm 9518
>> Silver Spring, MD 20910
>> 301-427-8149
>>
>>  Comments? Have I been of assistance? Please let my supervisors know. :~)
>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>
>>
>> On Tue, Sep 24, 2013 at 4:05 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>
>>>  Sure, no problem.  So if you take your new shapefile with modified
>>> columns and try to upload it to the original layer name 'communities_pgs',
>>> can you tell for sure that it does not overwrite data in the columns that
>>> haven't changed from the old version to the new?  It should be doing that
>>> at least, and discarding any new columns, as well as probably leaving any
>>> original columns that were removed in the new file empty.  At least that
>>> would be better than not overwriting anything at all.
>>>
>>> Micah
>>>
>>>
>>>
>>> On 9/24/2013 3:47 PM, John Kennedy - NOAA Affiliate wrote:
>>>
>>> If the name of the shape file is the same as one of the layers, then
>>> geoserver presents the old data. I took the shape file I'm attempting to
>>> load as communities_pgs and named it communities_pgs20130924 and it loads
>>> as expected with the new columns and data.
>>>
>>>  Thanks for all your help on this issue.
>>>
>>>  John F. Kennedy, PhD, CPG
>>>  GIS Specialist
>>>  Contractor: Caelum Corp.
>>> NOAA Fisheries Office of Science and Technology
>>> 1315 East-West Highway
>>> SSMC3 Rm 9518
>>> Silver Spring, MD 20910
>>> 301-427-8149
>>>
>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>> :~)
>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 3:32 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>>
>>>>  John,
>>>>
>>>> You may have hit on an issue with the way our system works.  Can you
>>>> tell if the attribute values that are being saved in the database are
>>>> coming from the new shapefile you just tried to upload?  Could you insert a
>>>> test value and try the upload again?  I just want to check that the new
>>>> data is actually being written when you specify update=overwrite.  It
>>>> should be, but it would be good to confirm.
>>>>
>>>> I don't actually know how GeoServer handles the overwrite option behind
>>>> the scenes.  I thought it might drop the underlying database table and
>>>> recreate, but it may not be.  That would be why the schema is being
>>>> preserved.  Best way to find out is for me to open a ticket with OpenGeo.
>>>> I'll do that later today.
>>>>
>>>> Micah
>>>>
>>>>
>>>> On 9/24/2013 2:59 PM, John Kennedy - NOAA Affiliate wrote:
>>>>
>>>> Okay, I now feel a little silly. I did not replace the dummy password
>>>> in my notes file with my LDAP password. Shape files and zip files have the
>>>> same names as the layer. I attempted the data load again, using --update
>>>> overwrite, and I got a 201 response. However, when I look at the layer in
>>>> geoserver I see the old data. There is a column (property) called
>>>> GEOID_jlf and that column is in the old data layer and not in the new
>>>> layer I'm attempting to load.
>>>>
>>>>  [image: Inline image 1]
>>>>
>>>>
>>>>  John F. Kennedy, PhD, CPG
>>>>  GIS Specialist
>>>>  Contractor: Caelum Corp.
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 9518
>>>> Silver Spring, MD 20910
>>>> 301-427-8149
>>>>
>>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>>> :~)
>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 2:35 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>>>
>>>>>  Ok.  Just curious, what are the shapefile names you're trying to
>>>>> upload?  Are they the communities_pts and communities_pgs layers?
>>>>>
>>>>> Looks like those are gone from the FeatureTypes list now (
>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html).
>>>>> But they still show up in the list when you go to the 'New Layer' page.
>>>>> Not sure I've seen that issue before, so might have to file a ticket.
>>>>>
>>>>> As for credentials, it should only be your NOAA LDAP username and
>>>>> password.  Those don't get passed on to the database, there's a separate
>>>>> way database authorization is handled.  If you can log in to the Web UI,
>>>>> there shouldn't be an issue with the REST API, or at least you shouldn't be
>>>>> getting that 401 response.
>>>>>
>>>>> Micah
>>>>>
>>>>>
>>>>> On 9/24/2013 2:21 PM, John Kennedy - NOAA Affiliate wrote:
>>>>>
>>>>> Thanks. Tomorrow works better for me. As for the credentials, nothing
>>>>> as changed on this end. I was able to load data with the hpcc_data.py
>>>>> script up to yesterday, but I did get an error message when attempting to
>>>>> overwrite the layer/feature type. Now, there maybe credential issues in the
>>>>> database (PostGIS?) but I've no way of knowing.
>>>>>
>>>>>  John F. Kennedy, PhD, CPG
>>>>>  GIS Specialist
>>>>>  Contractor: Caelum Corp.
>>>>> NOAA Fisheries Office of Science and Technology
>>>>> 1315 East-West Highway
>>>>> SSMC3 Rm 9518
>>>>> Silver Spring, MD 20910
>>>>> 301-427-8149
>>>>>
>>>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>>>> :~)
>>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>>>
>>>>>
>>>>> On Tue, Sep 24, 2013 at 2:15 PM, Micah Wengren <micah.wengren at noaa.gov
>>>>> > wrote:
>>>>>
>>>>>>  Hi John,
>>>>>>
>>>>>> That is odd for the 401 response on the data upload.  Are you sure
>>>>>> the credentials are right?  The other two seem ok, depending on if the
>>>>>> featuretype existed in the first place or not.
>>>>>>
>>>>>> I might be able to come by later, but right now I have a deadline to
>>>>>> finish something this afternoon.  Let me know if you aren't able to get it
>>>>>> to work.
>>>>>>
>>>>>> Micah
>>>>>>
>>>>>>
>>>>>> On 9/24/2013 12:32 PM, John Kennedy - NOAA Affiliate wrote:
>>>>>>
>>>>>> Hi Micah,
>>>>>>
>>>>>>  Well, I think I'm getting close. I installed pip, python setup, and
>>>>>> request. I used hpcc_date_requests.py to delete the layer and the feature
>>>>>> type and I get these to responses:
>>>>>> Response: 200, url=
>>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>>>
>>>>>>  Response: 404, url=
>>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>>>
>>>>>>  The hpcc_date_requests appears to be attempting to load data, but
>>>>>> the response back is this:
>>>>>> Response: 401, url=
>>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/file.shp?update=overwrite&configure=all
>>>>>>
>>>>>>  Any chance you can come to my cube and have a look?
>>>>>>
>>>>>>  Cheers,
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>>  John F. Kennedy, PhD, CPG
>>>>>>  GIS Specialist
>>>>>>  Contractor: Caelum Corp.
>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>> 1315 East-West Highway
>>>>>> SSMC3 Rm 9518
>>>>>> Silver Spring, MD 20910
>>>>>> 301-427-8149
>>>>>>
>>>>>>  Comments? Have I been of assistance? Please let my supervisors
>>>>>> know. :~)
>>>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov
>>>>>> >
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 24, 2013 at 10:18 AM, Micah Wengren <
>>>>>> micah.wengren at noaa.gov> wrote:
>>>>>>
>>>>>>>  Hi John,
>>>>>>>
>>>>>>> How are you doing the attempted data upload?  With the python
>>>>>>> script?  Can you tell me which python script you are using if so?  If you
>>>>>>> can, try using the hpcc_data_requests.py script instead of the other one
>>>>>>> (might involve some admin rights to install python modules on your
>>>>>>> workstation).  I've update that one more recently.  I'd like to drop the
>>>>>>> other one at some point for easier maintenance.
>>>>>>>
>>>>>>> I think you should be able to upload a new data set to replace the
>>>>>>> existing one, even if it already exists in the database and data store (but
>>>>>>> could be wrong if there's an issue going on here with these layers).
>>>>>>> Something like the following should work:
>>>>>>>
>>>>>>> > hpcc_data_requests.py -a data -u <name> -p <pass> -w nmfs_st -f
>>>>>>> /path/to/shapefile.zip --type shape --update overwrite  (the shapefile
>>>>>>> would need to be named to match the layer you want to update).
>>>>>>>
>>>>>>>
>>>>>>> Also, this is something that is a little whacky with the GeoServer
>>>>>>> REST API (and should be better documented somewhere admittedly), but you
>>>>>>> may need to do a delete command for both the Layer and the FeatureType to
>>>>>>> fully delete it.  When you create a new layer with the Web UI, it actually
>>>>>>> makes both the Layer and FeatureType behind the scenes.  Not sure if this
>>>>>>> will fix this particular issue, but commands to do that would be something
>>>>>>> like:
>>>>>>>
>>>>>>> > hpcc_data_requests.py -a delete_layer -u <name> -p <pass> -w
>>>>>>> nmfs_st --layername <layername>
>>>>>>> > hpcc_data_requests.py -a delete_featuretype -u <name> -p <pass> -w
>>>>>>> nmfs_st --layername <layername>
>>>>>>>
>>>>>>> This should fully delete both a layer and its corresponding
>>>>>>> featuretype.  It's possible if you did a delete_layer command with via the
>>>>>>> script without doing delete_featuretype as well, some featuretypes got left
>>>>>>> behind.  Again, an issue that should really be better documented in
>>>>>>> GeoServer docs but unfortunately I haven't seen it described anywhere, I
>>>>>>> happened to find out myself after having some similar issues.
>>>>>>>
>>>>>>> I'm looking around the workspace now, and I think I see some cases
>>>>>>> where there are FeatureTypes that aren't associated with any layers, in
>>>>>>> particular 'communities_pgs' and 'communities_pts' are both FeatureTypes
>>>>>>> without corresponding layers.  This could be preventing the data upload
>>>>>>> from happening for these layers.  I'm just looking at these two lists:
>>>>>>>
>>>>>>>
>>>>>>> https://services.ogc.noaa.gov/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage(filtered by 'nmfs_st')
>>>>>>>
>>>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html
>>>>>>>
>>>>>>> Also, you should probably do the delete_featuretype command for
>>>>>>> 'communities_pgs20130506' and 'communities_pts20130506', since they don't
>>>>>>> have corresponding Layers.  I think that will get those two in sync.
>>>>>>>
>>>>>>> I'm not sure why you get such a long list in the New Layer dialog
>>>>>>> though, looks like a bug to me.  No idea where 'SI_SE_Gulf_sites_pg_ALLsites'
>>>>>>> and 'SI_SE_Gulf_sites_pt' are coming from, because those aren't
>>>>>>> listed as current FeatureTypes in the REST API.
>>>>>>>
>>>>>>> HTH,
>>>>>>> Micah
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 9/24/2013 9:30 AM, John Kennedy - NOAA Affiliate wrote:
>>>>>>>
>>>>>>> The two layers of concern are communities_pgs and communities_pts.
>>>>>>> When I attempt to load new data, via a zip file with a shape file, I get a
>>>>>>> message that the communities_pgs exists. When I look at the list of source
>>>>>>> under the "Add sources" button I can see the communities_pgs layer and the
>>>>>>> publish link. But when I click the link and attempt to save I get the
>>>>>>> message below. Deleting layer via the hpcc_data python script also seems to
>>>>>>> fail.
>>>>>>>
>>>>>>>  [image: Inline image 1]
>>>>>>>
>>>>>>>  John F. Kennedy, PhD, CPG
>>>>>>>  GIS Specialist
>>>>>>>  Contractor: Caelum Corp.
>>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>>> 1315 East-West Highway
>>>>>>> SSMC3 Rm 9518
>>>>>>> Silver Spring, MD 20910
>>>>>>> 301-427-8149
>>>>>>>
>>>>>>>  Comments? Have I been of assistance? Please let my supervisors
>>>>>>> know. :~)
>>>>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>>>>  Hussain Jabalpurwala - NOAA Affiliate <
>>>>>>> hussain.jabalpurwala at noaa.gov>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 24, 2013 at 9:20 AM, Tim Haverland - NOAA Federal <
>>>>>>> tim.haverland at noaa.gov> wrote:
>>>>>>>
>>>>>>>> I'm adding John Kennedy to this thread so he can work with you on
>>>>>>>> this issue.
>>>>>>>>
>>>>>>>>  Tim
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 24, 2013 at 8:49 AM, Micah Wengren <
>>>>>>>> micah.wengren at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>>  Tim,
>>>>>>>>>
>>>>>>>>> I can see some layers in your workspace in the New Layer dialog
>>>>>>>>> that aren't currently published and have a 'Publish' link next to them, so
>>>>>>>>> I see what you mean.  Are you able to create a new SQL view layer with the
>>>>>>>>> same name as one of those layers, or does that not work?  Can you give me
>>>>>>>>> the name of one of the layers that you've tried with and I'll pass it on to
>>>>>>>>> OpenGeo support to troubleshoot, that way I can give them a specific layer
>>>>>>>>> name with the issue.
>>>>>>>>>
>>>>>>>>> The layers should be disappearing but for whatever reason they're
>>>>>>>>> not.
>>>>>>>>>
>>>>>>>>> Micah
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 9/23/2013 3:55 PM, Tim Haverland - NOAA Federal wrote:
>>>>>>>>>
>>>>>>>>>  Hi all,
>>>>>>>>>
>>>>>>>>>  We've been trying to clean up a bunch of layers in our
>>>>>>>>> workspace. We can delete layers using the "Remove selected resources"
>>>>>>>>> button on the Layers page, but when we try to replace that layer with
>>>>>>>>> another, the layer is still there.
>>>>>>>>>
>>>>>>>>>  We can verify this by clicking "Add a new resource" ... all of
>>>>>>>>> the old layers are still there and I can click Publish to make them "live"
>>>>>>>>> again.
>>>>>>>>>
>>>>>>>>>  How do we permanently get rid of unwanted layers?
>>>>>>>>>
>>>>>>>>>  Tim
>>>>>>>>> --
>>>>>>>>> *Tim Haverland*
>>>>>>>>> Acting Operations Branch Chief
>>>>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>>>>> 1315 East-West Highway
>>>>>>>>> SSMC3 Rm 12303
>>>>>>>>> Silver Spring, MD 20910
>>>>>>>>> 301-427-8137
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  _______________________________________________
>>>>>>>>> Open.ogc mailing listOpen.ogc at list.woc.noaa.govhttps://list.woc.noaa.gov/cgi-bin/mailman/listinfo/open.ogc
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  --
>>>>>>>> *Tim Haverland*
>>>>>>>> Acting Operations Branch Chief
>>>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>>>> 1315 East-West Highway
>>>>>>>> SSMC3 Rm 12303
>>>>>>>> Silver Spring, MD 20910
>>>>>>>> 301-427-8137
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/a9e1b059/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 153411 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/a9e1b059/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 164721 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/a9e1b059/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166250 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/a9e1b059/attachment-0005.png>

More information about the Open.ogc mailing list